Solution Overview

Application Scenarios

This solution enables one-click deployment of the LiteLLM unified LLM API gateway on Huawei Cloud Flexus X instances. LiteLLM unifies 100+ LLM providers (OpenAI, Anthropic, Cohere, etc.) into a single OpenAI-compatible API, with built-in virtual key management, spending tracking, load balancing, and an admin dashboard.

This solution is typically applied in the following scenarios:

• Unified LLM Management for Multiple Business Lines: Leverage LiteLLM's virtual key mechanism to create independent keys for each team/project, set budget caps, and automatically track usage.
• High-Availability LLM Infrastructure: For enterprises integrating AI into core business processes and requiring SLA guarantees.
• Personal AI Management System: For individual developers using OpenAI-compatible APIs to switch between multiple providers during AI development and testing.

Architecture

This solution helps you quickly deploy the LiteLLM unified LLM API gateway.

Single-Cloud Server Deployment

Figure 1 Architecture

This Single-Cloud Server Deployment solution will:

• Create a FlexusX instance(Elastic Cloud Server (ECS)) to set up the LiteLLM.

• Create one Elastic IP (EIP) for internal and external communication.
• Create a security group and configure security group rules to protect Huawei Cloud cloud servers.

HA Deployment in CCE Container

Figure 2 Architecture

HA deployment in CCE container for LiteLLM:

• Create three EIPs to enable access to and from the Internet.
• Create one ELB and associate an EIP with it to automatically distribute access traffic to various backend services. This enhances the external service capabilities of the application system and achieves robust application fault tolerance.
• Create one NAT gateway and bind an EIP to it. Add SNAT rules to let all instances in a private subnet connect outbound to the Internet while remaining fully private.
• Create three ECS instances for deploying LiteLLM Proxy core components and services.
• Create one CCE Turbo cluster, create a node pool, and manage the three ECS instances as nodes in the cluster.
• Create one Distributed Cache Service (DCS) for Redis instance that is compatible with Redis, providing users with high-performance, cost-effective NoSQL databases while ensuring data consistency during data circulation (for global rate limiting and cache synchronization).
• Create one RDS for PostgreSQL instance with primary/standby deployment across AZs to ensure disaster recovery capabilities (for storing api keys and spend log metadata).
• Create four security groups with specified rules to protect cloud services.

Advantages

• Unifies 100+ LLM providers with OpenAI API compatibility.
• Virtual key management with per-key budget caps and real-time usage tracking.
• Multi-provider failover and load balancing.

Constraints

• Before deploying this solution, ensure you have registered a Huawei account, activated Huawei Cloud, completed real-name verification, and that your account is not in arrears or frozen. If selecting the "Yearly/Monthly" billing mode, ensure sufficient account balance for automatic payment during one-click deployment, or manually pay via the "Pending Orders" section in the Cost Center during deployment.
• If using IAM delegated permissions for resource deployment, ensure your Huawei Cloud account has sufficient IAM permissions (refer to creating the rf_admin_trust delegation). If using a primary account or an IAM sub-account under the "admin" group, delegation is optional, and deployment will use the permissions of the currently logged-in user.
• This content may contain technical descriptions and configuration examples of third-party open-source software, components, or frameworks. You are required to independently evaluate and decide whether to use them. Huawei Cloud makes no warranties regarding the functionality, performance, security, compatibility, or compliance of any third-party software. Intellectual property rights of third-party software belong to their respective owners. You must strictly comply with applicable open-source license terms (such as MIT, Apache 2.0, GPL, etc.), ensure that no third-party legitimate rights are infringed, and take responsibility for security evaluations and vulnerability management. Huawei Cloud and its affiliates shall not be liable for any direct or indirect losses (including but not limited to data loss, business interruption, security incidents, intellectual property disputes, compliance penalties, etc.) arising from your use of any third-party software mentioned in this solution. References to third-party software in this solution do not constitute endorsement or approval, nor do they imply any authorization or partnership between Huawei Cloud and third-party rights holders.