Accessing a Real-Time Service Through ELB

Context

ELB automatically distributes incoming traffic across multiple backend servers based on configured listening rules. It expands the service capabilities of your applications and improves their availability by eliminating individual faults.

ModelArts allows external users to securely and quickly access its real-time services and easily integrate prediction services. It uses ELB for private network access. ELB handles incoming traffic and sends it to the backend inference service instances based on the forwarding rules.

Billing

Using ELB and VPC endpoints is billed. For details, see Elastic Load Balance Billing and VPC Endpoint Billing.

Step 1: Creating an ELB

Step 2: Configuring ELB Access in a ModelArts Real-Time Inference Service

When configuring the inference service information, configure the ELB parameters. For details, see Configuring Service Information.

The ELB parameters are as follows:

1. Log in to the ModelArts console. In the navigation pane, choose Model Inference > Real-Time Inference.
2. Click Deploy. The Deploy Real-Time Service page is displayed. When configuring the inference service information, select the created load balancer under ELB (Optional).
   Figure 1 Selecting a load balancer
   

3. After selecting a load balancer, if the service uses HTTPS or WSS, configure the SSL parsing mode. Choose either unidirectional or bidirectional authentication. For unidirectional authentication, configure only the server certificate. For bidirectional authentication, configure both the server and CA certificates. In addition, you can configure SNI.
   Figure 2 One-way authentication
   
   Figure 3 Two-way authentication
   

   The CA certificate, server certificate, and SNI certificate must be created in advance. For details, see Adding a Certificate. For more information about SNI, see Using SNI Certificates for Access Through Multiple Domain Names.

   For details about other parameters for service information and deployment configuration, see Configuring Service Information and Deploying a Real-Time Inference Service Using a Single Node. After you finish the deployment setup, follow these steps to access and use the service.

Step 3: Accessing the Real-Time Service Using the ELB Address

After configuring an ELB for the service and deploying the service, you can click the service in the real-time service list or on the service details page to view the ELB access address.

Figure 4 Calling a service

The service call information will display the private or public network call URLs one by one based on the IP addresses of the ELB.

Figure 5 Calling information

The format of the ELB call address for real-time services is: ${service_protocol}://${ELB_instance_IP}/v2/infer/${real-time_service_ID}/${specific_API}. For example: http://10.10.xx.xxx/v2/infer/d64cda4f-af92-4bd3-804d-xxxxx/v1/chat/completions.

Since users can change the private IP address or EIP bound to an ELB instance at any time via the ELB console or ELB APIs, a Synchronize button is provided at the bottom of the real-time service call information section. This allows one-click synchronization of the latest ELB IP address to ensure that the real-time service remains accessible via the new IP address.

If you have already set up other network components that handle Layer-4 traffic before creating the real-time service, click Custom URL in the Call Information and add the IP address or domain name of the network component. This allows you to access the real-time service using the IP address or domain name. For example, if you have also connected APIG before configuring the ELB and wants to access the real-time service via the APIG domain name, you need to add the APIG domain name to the custom addresses.

Figure 6 Modifying the access address

VPC Endpoint Creation Rules

If the VPC associated with the ELB accessed by you is connecting for the first time, a VPC endpoint will be automatically created under this VPC to bridge the network between this VPC and the real-time service. The IP address of the VPC endpoint will be used as the backend server of the ELB.

If the VPC endpoint automatically created by the ModelArts real-time service is no longer used, it will be automatically deleted.

ELB Resources Creation Rules

When you access an ELB, a series of related resources will be created under the ELB, including listeners, backend server groups, and forwarding policies. Specifically:

• If the service protocol is HTTP, the ELB creates an HTTP listener on port 80. If the service protocol is HTTPS, the ELB creates an HTTPS listener on port 443.
• If the service runs in synchronous mode, a rule with the path /v2/infer forwards calls to the synchronous real-time service. If the service runs in asynchronous mode, a rule with the path /v2/async-infer forwards calls to the asynchronous real-time service.

If the above ELB-related resources already exist, they will be reused; if they do not exist, they will be created automatically. During the reuse process, configuration conflicts may occur. Common configuration conflicts are as follows:

1. Listeners: If the service protocol is HTTP, but the ELB is already bound to a listener with a different protocol (e.g., HTTPS, TCP) using port 80, it is considered a configuration conflict and ELB access will fail.
2. Forwarding policies: If the service runs in synchronous mode, and the listener already contains a forwarding rule with a path prefix matching /v2/infer whose forwarding target is not a backend server group, or whose backend server in the group is not the target VPC endpoint IP, it is considered a configuration conflict and ELB access will fail.

If an ELB automatically created by the ModelArts real-time service is no longer in use, the aforementioned ELB-related resources will be automatically cleaned up.

Checking the ELB Interconnection Status

On the inference service details page, view the ELB interconnection status in the network configuration area. If the ELB fails to be connected or disconnected, you can retry on the inference details page.

Figure 7 Checking the ELB interconnection status