Solution Overview

Application Scenarios

This solution enables one-click deployment of the LiteLLM unified LLM API gateway on Huawei Cloud Flexus X instances. LiteLLM unifies 100+ LLM providers (OpenAI, Anthropic, Cohere, etc.) into a single OpenAI-compatible API, with built-in virtual key management, spending tracking, load balancing, and an admin dashboard.

This solution is typically applied in the following scenarios:

• Unified LLM Management for Multiple Business Lines: Leverage LiteLLM's virtual key mechanism to create independent keys for each team/project, set budget caps, and automatically track usage.
• High-Availability LLM Infrastructure: For enterprises integrating AI into core business processes and requiring SLA guarantees.
• Personal AI Management System: For individual developers using OpenAI-compatible APIs to switch between multiple providers during AI development and testing.

Architecture

This solution helps you quickly deploy the LiteLLM unified LLM API gateway.

Single-Cloud Server Deployment

Figure 1 Architecture

This Single-Cloud Server Deployment solution will:

• Create one Virtual Private Cloud (VPC)and subnet for the network infrastructure required to build cloud services.
• Create a Flexus X Instance (FlexusX)(Elastic Cloud Server (ECS)) to set up the LiteLLM.

• Create one Elastic IP (EIP) for internal and external communication.
• Create a security group and configure security group rules to protect Huawei Cloud cloud servers.

HA Deployment in CCE Container

Figure 2 Architecture

HA deployment in CCE container for LiteLLM:

• Create one Virtual Private Cloud (VPC)and subnet for the network infrastructure required to build cloud services.
• Create three Elastic IP (EIP) to enable access to and from the Internet.
• Create one Elastic Load Balance (ELB) and associate an EIP with it to automatically distribute access traffic to various backend services. This enhances the external service capabilities of the application system and achieves robust application fault tolerance.
• Create one NAT Gateway and bind an EIP to it. Add SNAT rules to let all instances in a private subnet connect outbound to the Internet while remaining fully private.
• Create three Elastic Cloud Server (ECS) instances for deploying LiteLLM Proxy core components and services.
• Create one Cloud Container Engine CCE Turbo cluster, create a node pool, and manage the three ECS instances as nodes in the cluster.
• Create one Distributed Cache Service (DCS) for Redis instance that is compatible with Redis, providing users with high-performance, cost-effective NoSQL databases while ensuring data consistency during data circulation (for global rate limiting and cache synchronization).
• Create one RDS for PostgreSQL instance with primary/standby deployment across AZs to ensure disaster recovery capabilities (for storing api keys and spend log metadata).
• Create four security groups with specified rules to protect cloud services.

Advantages

• Unifies 100+ LLM providers with OpenAI API compatibility.
• Virtual key management with per-key budget caps and real-time usage tracking.
• Multi-provider failover and load balancing.

Constraints

• Before deploying this solution, you must register a Huawei account, activate Huawei Cloud, complete real-name authentication, and ensure the account is not in arrears or frozen. If you select the "prepaid" billing mode, ensure sufficient account balance for automatic payment during one-click resource deployment; alternatively, manually complete payment via "Unpaid Orders" in the Billing Center during the one-click deployment process.
• If using IAM agency permissions to deploy resources, ensure your Huawei Cloud account has sufficient IAM permissions. For details, refer to creating the rf_admin_trust agency. If deploying with a Huawei primary account or an IAM sub-account under the admin user group, you may skip the agency option, and the deployment will proceed using the permissions of the currently logged-in user.
• This content may include technical descriptions and configuration examples of third-party open-source software, components, or frameworks. You should independently evaluate and decide whether to use them. Huawei Cloud makes no warranties regarding the functionality, performance, security, compatibility, or compliance of third-party software. Intellectual property rights of third-party software belong to their respective owners. You must strictly comply with applicable open-source license terms (such as MIT, Apache 2.0, etc.), ensure no infringement of any third-party rights, and take full responsibility for security assessments and vulnerability management. Huawei Cloud and its affiliates shall not be liable for any direct or indirect losses (including but not limited to data loss, business interruption, security incidents, intellectual property disputes, compliance penalties, etc.) arising from your use of any third-party software mentioned in this solution. References to third-party software in this solution do not constitute endorsement or imply any authorization or partnership between Huawei Cloud and the third-party rights holders.