Solution Overview

Application Scenarios

This solution helps you quickly deploy the Community Standalone Edition and CCE Container High Availability Edition New API Large Model Gateway. New API is an open-source project based on the secondary development of One API, focusing on AI model interface management and distribution systems. It supports converting various LLM into OpenAI format for invocation and is compatible with interfaces such as Midjourney Proxy, Suno, and Rerank.

Architecture

This solution helps you quickly deploy New API LLM gateway.

Figure 1 Architecture (Community Standalone Edition)

Figure 2 Architecture (CCE Container High Availability Edition)

This solution will:

Community Standalone Edition:

• Create three Flexus X Instance (FlexusX) : Used to deploy the New API LLM gateway container cluster.
• Create one Elastic IP (EIP) : Provides capabilities for accessing the public network and being accessed from the public network.

CCE Container High Availability Edition:

• Create one Cloud Container Engine CCE Turbo cluster: Used to deploy the New API LLM gateway container cluster.
• Create three Flexus X Instance (FlexusX) : Serves as the worker nodes for the CCE container cluster to host container instances.
• Create one Elastic IP (EIP) : Provides capabilities for accessing the public network and being accessed from the public network.
• Create one Elastic Load Balance (ELB) : Automatically distributes external inference traffic, enhancing external service capability and application fault tolerance.
• Create one NAT Gateway : Allows instances within private subnets and container subnets to securely initiate outbound Internet connections to request upstream APIs.
• Create one Distributed Cache Service (DCS): Used for global rate limiting and route cache status synchronization across multiple replicas.
• Create one RDS for MySQL: Used for the persistent storage of New API user API keys, organizational permissions, and routing metadata.

Advantages

• One-Click Deployment

  Easily deploy with one click to complete the creation of cloud service resources and the setup of the New API large model gateway.

• Ready to Use

  New API adopts a modern microservices architecture, emphasizing high availability and ease of deployment. It is suitable for content creation, education, scientific research, and other fields, offering ready to use functionality after deployment.

• Cost Optimization

  Provides cost-effective cloud servers with on-demand resource selection, supports auto-scaling to reduce idle resources, optimizes cost investment, and further lowers customer operational expenses.

Constraints

• Before deploying this solution, ensure that you have created a Huawei ID with access to the target region and enabled Huawei Cloud services. If you select the yearly/monthly billing mode, ensure that your account has sufficient balance. If you do not have sufficient balance, you can go to the Billing Center to manually pay for the order.
• If using IAM agency permissions to deploy resources, ensure your Huawei Cloud account has sufficient IAM permissions. For details, refer to creating the rf_admin_trust agency. If deploying with a Huawei primary account or an IAM sub-account under the admin user group, you may skip the agency option, and the deployment will proceed using the permissions of the currently logged-in user.
• This content may include technical descriptions and configuration examples of third-party open-source software, components, or frameworks. You should independently evaluate and decide whether to use them. Huawei Cloud makes no warranties regarding the functionality, performance, security, compatibility, or compliance of third-party software. Intellectual property rights of third-party software belong to their respective owners. You must strictly comply with applicable open-source license terms (such as MIT, Apache 2.0, etc.), ensure no infringement of any third-party rights, and take full responsibility for security assessments and vulnerability management. Huawei Cloud and its affiliates shall not be liable for any direct or indirect losses (including but not limited to data loss, business interruption, security incidents, intellectual property disputes, compliance penalties, etc.) arising from your use of any third-party software mentioned in this solution. References to third-party software in this solution do not constitute endorsement or imply any authorization or partnership between Huawei Cloud and the third-party rights holders.