Solution Overview

Application Scenarios

This solution helps you quickly deploy the community standalone version and the CCE container high-availability version of the New API large model gateway. New API is a large model API aggregation and distribution tool that consolidates Ollama and various cloud model channels into a single backend, providing a relatively unified OpenAI-compatible interface externally, while managing users, tokens, quotas, and call logs.

Architecture

This solution helps you quickly deploy New API LLM gateway.

Figure 1 Architecture (Community Standalone Edition)

Figure 2 Architecture (CCE Container High Availability Edition)

This solution will:

Community Standalone Edition:

• Create three Flexus X Instance (FlexusX) : Used to deploy the New API LLM gateway container cluster.
• Create one Elastic IP (EIP) : Provides capabilities for accessing the public network and being accessed from the public network.

CCE Container High Availability Edition:

• Create one Cloud Container Engine CCE Turbo cluster: Used to deploy the New API LLM gateway container cluster.
• Create three Flexus X Instance (FlexusX) : Serves as the worker nodes for the CCE container cluster to host container instances.
• Create one Elastic IP (EIP) : Provides capabilities for accessing the public network and being accessed from the public network.
• Create one Elastic Load Balance (ELB) : Automatically distributes external inference traffic, enhancing external service capability and application fault tolerance.
• Create one NAT Gateway : Allows instances within private subnets and container subnets to securely initiate outbound Internet connections to request upstream APIs.
• Create one Distributed Cache Service (DCS): Used for global rate limiting and route cache status synchronization across multiple replicas.
• Create one RDS for MySQL: Used for the persistent storage of New API user API keys, organizational permissions, and routing metadata.

Advantages

• One-Click Deployment

  Easily deploy with one click to complete the creation of cloud service resources and the setup of the New API large model gateway.

• Ready to Use

  New API adopts a modern microservices architecture, emphasizing high availability and ease of deployment. It is suitable for content creation, education, scientific research, and other fields, offering ready to use functionality after deployment.

• Cost Optimization

  Provides cost-effective cloud servers with on-demand resource selection, supports auto-scaling to reduce idle resources, optimizes cost investment, and further lowers customer operational expenses.

Constraints

• Before deploying this solution, you must register a Huawei account, activate Huawei Cloud, complete real-name authentication, and ensure the account is not in arrears or frozen. If you select the "prepaid" billing mode, ensure sufficient account balance for automatic payment during one-click resource deployment; alternatively, manually complete payment via "Unpaid Orders" in the Billing Center during the one-click deployment process.
• If using IAM agency permissions to deploy resources, ensure your Huawei Cloud account has sufficient IAM permissions. For details, refer to creating the rf_admin_trust agency. If deploying with a Huawei primary account or an IAM sub-account under the admin user group, you may skip the agency option, and the deployment will proceed using the permissions of the currently logged-in user.
• This content may include technical descriptions and configuration examples of third-party open-source software, components, or frameworks. You should independently evaluate and decide whether to use them. Huawei Cloud makes no warranties regarding the functionality, performance, security, compatibility, or compliance of third-party software. Intellectual property rights of third-party software belong to their respective owners. You must strictly comply with applicable open-source license terms (such as MIT, Apache 2.0, etc.), ensure no infringement of any third-party rights, and take full responsibility for security assessments and vulnerability management. Huawei Cloud and its affiliates shall not be liable for any direct or indirect losses (including but not limited to data loss, business interruption, security incidents, intellectual property disputes, compliance penalties, etc.) arising from your use of any third-party software mentioned in this solution. References to third-party software in this solution do not constitute endorsement or imply any authorization or partnership between Huawei Cloud and the third-party rights holders.