Migrating from Old-Version Real-Time Services to New-Version Real-Time Services

Scenarios

ModelArts real-time services (old-version) will soon be discontinued. For AI model deployment scenarios, users currently using the old version may experience service interruptions due to its discontinuation. The new version of ModelArts real-time services supports all large model formats and distributed deployment, providing more comprehensive metric monitoring and a smoother O&M experience. Consequently, smoothly migrating old-version services to the new version has become a common user requirement. This section describes how to create corresponding inference services on the new Real-Time Services page based on existing configurations and model image paths.

Overview

The new-version real-time services feature an evolved architecture, introducing KubeInfer, a brand-new distributed inference solution. By leveraging multi-node parallel computing, distributed inference significantly enhances the efficiency of large-scale models. Through architecture compatibility design, the new version supports distributed inference while maintaining full compatibility with single-node inference scenarios. Therefore, you can deploy models and images from the old-version services directly on the new Real-Time Services page.

Permissions

You must have the permission set for the new inference version. For details, see Configuring Basic Permissions for Inference Deployment.

Billing

New-version real-time services currently can only be deployed in dedicated resource pools. Billing will be incurred for compute resources in these pools, as detailed in Table 1.

When you create a dedicated resource pool, the billing starts after the node is created. The driver upgrade and installation process is also considered as a part of the creation success.

Preparations

Prepare a dedicated resource pool whose job type is model deployment. You can purchase a dedicated resource pool or change the job type of an existing dedicated resource pool.

• Purchase a dedicated resource pool: Select Model Deployment as the job type. For details, see Creating a Dedicated Resource Pool.
• Modify job types of a dedicated resource pool: Add the Model Deployment option to an existing pool's job types. For details, see Modifying the Job Types Supported by a Dedicated Resource Pool.

  If your old-version inference services are still running, do not deselect the Model Deploy (Old Version) job type. You may deselect it only after your service migration is complete.

Step 1: Obtaining the Deployment Information of Old-Version Real-Time Services

1. Log in to the ModelArts console and choose Model Deployment > Real-Time Services (Old Version).
2. Click the name of the service to be migrated. The service details page is displayed.
3. In the Configuration Updates tab, click the model name in the model list to view the model details and obtain the configuration information about the model image.
   Figure 1 Model details
   

   Table 2 describes the mapping of model and deployment parameters between the new and old versions of real-time services.

   Table 2 Parameter mapping between the new and old versions of real-time services

   Old-Version Real-Time Service Parameter	Parameter Value	New-Version Real-Time Service Parameter	Parameter Value
   Meta Model Source	Training job	Deployment Configuration > Model Settings > Model Source	Model Source: Select Custom Model. Storage Type: Select Object Storage Service – Bucket. Storage Location: Enter the output path of the training job. Go to the training job details page on the console to obtain the output path. For details, see Viewing Training Jobs and Details. Mount Path: Set it based on the new service. Local Storage Acceleration: Enable this function based on new business needs.
   	OBS		Model Source: Select Custom Model. Storage Type: Select Object Storage Service – Bucket. Storage Location: Set it to the same path as the storage path of the meta model for the old real-time service. Mount Path: Set it based on the new service. Local Storage Acceleration: Configure this parameter based on new business needs.
   	Container image	Deployment Configuration > Unit Settings > Image Type	No model configuration required; configure directly in the image settings. Image Type: Select Custom Images. Image: Same as the Container Image Storage Path in the old version.
   AI Engine	TensorFlow, PyTorch, MindSpore	Deployment Configuration > Unit Settings > Image Type	Image Type: Select Preset Images. Image: Select a preset image of the required framework.
   	Custom		Image Type: Select Custom Images. Image: Same as the Container Image Storage Path in the old version.
   Container API (custom AI engine)	Protocol and port number.	Deployment Configuration > Deployment Management Settings > Container Port	Same as the old version.
   Health Check (custom AI engine)	Startup, Readiness, and Liveness probes.	Deployment Configuration > Unit Settings > Health Check	Same as the old version.
   Model Description	Document name and URL.	Deployment Configuration > Basic Information > Description (Optional)	Use the service description to include model details.
   Instruction Set Architecture	System architecture of the model.	/	Not maintained; you must ensure the correctness of the runtime architecture.
   Inference Accelerator	Enable/Disable accelerator cards.	/	Not maintained; you must ensure the correctness of the accelerator cards.
   Deployment Type	Real-time service	/	Simply create the service on the Model Deployment > Real-Time Services page.
   Boot Command	Custom boot command.	Deployment Configuration > Unit Settings > More Settings > Boot Command	Same as the old version.

4. Return to the Real-Time Services (Old Version) list page, click Modify in the Operation column for the service you want to migrate, and enter the modification page to obtain its deployment information.
   Table 3 describes the mapping of model and deployment parameters between the new and old versions of real-time services.

   Table 3 Parameter mapping between the new and old versions of real-time services

   Old-Version Real-Time Service Parameter	Parameter Value	New-Version Real-Time Service Parameter	Parameter Value
   Auto Stop	Enable or disable the switch.	Service Information > Basic Information > Auto Stop	Select or deselect.
   Resource Pool	Public/Dedicated Resource Pool.	Deployment Configuration > Resource Settings > Resource Pool	The new version currently only supports dedicated resource pools. Select the pool prepared in Preparations.
   Multi-Pool Load Balancing	Enable or disable the switch.	/	After creating the service, you can add multiple deployments across different pools. For details, see Configuring Deployment Settings.
   Model and Configuration > Model Source	My Subscriptions.	/	Not supported.
   Model and Configuration > Model Source	My Model & Model and Version.	Deployment Configuration > Model Settings > Model Source Deployment Configuration > Unit Settings > Image Type	Set this parameter based on Meta Model Source in Table 2.
   Model and Configuration > Traffic Ratio (%)	Custom value.	After creating a service, add multiple deployments and configure different traffic percentages for them. For details, see Traffic Policies.	Same as the old version.
   Model and Configuration > Instance Flavor	Select an instance flavor from the drop-down list.	Deployment Configuration > Unit Settings > Unit Instance Specifications	Same as the old version.
   Model and Configuration > Node Affinity Deployment	Enable this function and choose the preferred nodes for deployment.	Deployment Configuration > Unit Settings > More Settings > Affinity Scheduling	Set affinity type and strength; select target nodes. Same as the old version.
   Model and Configuration > Instances	Custom value.	Deployment Configuration > Resource Settings > Deployment Replicas	Same as the old version.
   Model and Configuration > Environment Variable	Input key-value pairs.	Deployment Configuration > Unit Settings > Environment Variables	Same as the old version.
   Model and Configuration > Timeout	Custom value.	Deployment Configuration > Deployment Management Settings> More Settings > Deployment Timeout (Minutes)	Same as the old version.
   Model and Configuration > Mount Storage	Storage volume type: OBS bucket/OBS parallel file system Source address Mount path	Storing model or code files: Deployment Configuration > Unit Settings > Mount File Storage	Storage Type: Matches Volume Type of the old version. Storage Location: Matches the source address of the old version. Mount Path: Matches Mount Path of the old version.
   	Storage volume type: SFS Turbo Source address Mount path		Storage Type: Matches Volume Type of the old version. File System: Matches File System Name of the old version. Mount Path: Matches Mount Path of the old version.
   Traffic Limit	Requests per second.	Service Information > High Availability Settings > Rate Limiting Policy > Requests Per Second	Same as the old version.
   WebSocket	Enable or disable the switch.	Service Information > Network Settings > Service Protocol	Select WSS in the new version if this function is enabled in the old version.
   Advanced Log Management	Enable or disable the switch.	Service Information > Advanced Settings > Ingest Logs to LTS	Same as the old version.
   Log Dump	Enable or disable the switch.	Service Information > Advanced Settings > Ingest Logs to LTS	Same as the old version.
   Application Authentication	Enable or disable the switch.	Service Information > Network Settings > Authentication Mode	Not supported. You can select API key authentication in the service information.
   Notification	Enable or disable the switch.	/	Not supported.
   Advanced Configuration > Tags	Enter tag keys and values.	Service Information > Advanced Settings > Tags	Add tags. The tag keys and values must be the same as those of the old-version service.
   Advanced Configuration > IPv6 Support	Enable or disable the switch.	/	Not supported.

Step 2: Checking the Image

If rank_table information is used for service orchestration in the service image of the old inference version, the information is generated by the Modelarts-Infers-Operator component in the new inference version.

status

server_group_count

group_id

server_count

server_id

server_ip

pod_name

device

device_id

device_ip

device_logical_id

rank_id

{
    "version": "1.0",
    "status": "completed",
    "server_group_count": "2",
    "server_group_list": [
        {
            "group_id": "0",
            "server_count": "1",
            "server_list": [
                {
                    "server_id": "192.xxx.xxx.238",
                    "server_ip": "172.xxx.xxx.139",
                    "pod_name": "infer-xxx-role-0-xxx",
                    "device": [
                        {"device_id": "0", "device_ip": "29.xxx.xxx.197", "rank_id": "0"},
                        ...
                    ]
                }
            ]
        },
        {
            "group_id": "1",
            "server_count": "2",
            "server_list": [
                {
                    "server_id": "192.xxx.xxx.0",
                    "server_ip": "172.xxx.xxx.54",
                    "pod_name": "infer-xxx-role-1-xxx",
                    "device": [
                        {"device_id": "0", "device_ip": "29.xxx.xxx.74", "rank_id": "0"},
                        ...
                    ]
                },
                ...
            ]
        }
    ]
}

When using the rank table, wait until the status is completed. Reference script rank_table_checker.py:

import json
import os
import time

GLOBAL_RANK_TABLE_ENV = 'GLOBAL_RANK_TABLE_FILE_PATH'
POD_IP_ENV = 'POD_IP'

def wait_completed_global_rank_table():
    while True:
        try:
            pod_ip = os.getenv(POD_IP_ENV)
            global_rank_table_path = os.getenv(GLOBAL_RANK_TABLE_ENV)
            if not global_rank_table_path:
                print('read env \"{}\" failed'.format(GLOBAL_RANK_TABLE_ENV))
            with open(global_rank_table_path, 'r') as file:
                buf = file.read()
            rank_table = json.loads(buf)
            if rank_table["status"] == "completed":
                server_group_list = rank_table['server_group_list']
                for group in server_group_list:
                    server_list = group["server_list"]
                    for i in range(len(server_list)):
                        if server_list[i]["server_ip"] == pod_ip:
                            return
                print("cannot find local ip in ranktable!")
            else:
                print("status of ranktable is not completed!")
        except Exception as e:
            print(e)
        time.sleep(1)

if __name__ == "__main__":
    wait_completed_global_rank_table()

Step 3: Creating a New-Version Real-Time Service

1. Log in to the ModelArts console and choose Model Deployment > Real-Time Services.
2. In the real-time service list, click Deploy.
3. On the Deploy Real-Time Service page, configure the service and deployment information of the new-version real-time service based on the old-version settings in Table 2 and Table 3. For details about more parameters, see User Guide > "Deploying a Model as a Real-Time Service."
4. On the Deploy Real-Time Service > Confirmation page, confirm the configuration information and click Confirm Deployment.

   Deploying a service generally requires a period of time, which may be several minutes or tens of minutes depending on the amount of your data and resources.

   You can go to the real-time service list to check if the deployment is complete. Once the service status changes from Deploying to Running, the service is deployed.

Follow-Up Operations

After the deployment is complete, you can verify the service on the prediction page. For details, see Real-Time Service Prediction.

If the service works correctly, follow these guides to set up public network access: Accessing a Real-Time Service via Private Network through Dedicated APIG, WAF, VPC, and ELB and Accessing a Real-Time Services via Private Network through Load Balancing.