Help Center> ModelArts> SDK Reference> Service Management> Deploying a Real-Time Service

Updated on 2024-03-21 GMT+08:00

View PDF

Deploying a Real-Time Service

Real-time service deployment covers the following aspects:

Initialize a real-time service.
Deploy a real-time service predictor.
Deploy a batch service transformer.

The service object predictor is returned after deployment. The attributes of the service object include all functions described in this chapter.

Sample Code

In ModelArts notebook, you do not need to enter authentication parameters for session authentication. For details about session authentication of other development environments, see Session Authentication.

Method 1: Initialize the predictor that has been deployed as a real-time service.

      
           from modelarts.session import Session
from modelarts.model import Predictor

session = Session()
predictor_instance = Predictor(session, service_id="your_service_id")

Method 2: Deploy a real-time service predictor.

Deploy the service in a public resource pool.

        
         
           
           
             from modelarts.session import Session
from modelarts.model import Model
from modelarts.config.model_config import ServiceConfig, TransformerConfig, Schedule

session = Session()
model_instance = Model(session, model_id='your_model_id')
vpc_id = None                                        # (Optional) ID of the VPC where the real-time service instance is deployed. This parameter is left blank by default.
subnet_network_id = None                             # (Optional) Subnet ID. This parameter is left blank by default.
security_group_id = None                             # (Optional) Security group. This parameter is left blank by default.
configs = [ServiceConfig(model_id=model_instance.model_id,
                         weight="100",
                         instance_count=1,
                         specification="modelarts.vm.cpu.2u")]  # For details, see specification.
predictor_instance = model_instance.deploy_predictor(
            service_name="service_predictor_name",
            infer_type="real-time",
            vpc_id=vpc_id,
            subnet_network_id=subnet_network_id,
            security_group_id=security_group_id,
            configs=configs,                       # predictor configuration parameter. For details, see configs.
            schedule = [Schedule(op_type='stop', time_unit='HOURS', duration=1)]       # (Optional) Specify the runtime duration for a real-time service.
)

            

          

        
       

The model_id parameter specifies the model that is to be deployed as a real-time service. Obtain the value by calling the API described in Obtaining Models or from the ModelArts management console.

Deploy the service in a dedicated resource pool.

from modelarts.config.model_config import ServiceConfig

configs = [ServiceConfig(model_id=model_instance.model_id, weight="100", instance_count=1, 
						 specification="modelarts.vm.cpu.2u")]
predictor_instance = model_instance.deploy_predictor( 
                                                      service_name="your_service_name",
                                                      infer_type="real-time",
                                                      configs=configs,
                                                      cluster_id="your dedicated pool id"
                                                    )

configs is defined by ServiceConfig in the SDK. The type of configs is list, and the tuple object in the list is ServiceConfig. The code is as follows:

      
           configs = []
envs = {"model_name":"mxnet-model-1", "load_epoch":"0"}

service_config1 = ServiceConfig(
        model_id="model_id1",                 # model_id1 and model_id2 must be the IDs of different versions of the same model.
        weight="70",
        specification="modelarts.vm.cpu.2u",  # For details, see specification.
        instance_count=2,
        envs=envs)                            # (Optional) Configure the environment variable, for example, envs = {"model_name":"mxnet-model-1", "load_epoch":"0"}.
service_config2 = ServiceConfig(
        model_id='model_id2',
        weight="30",
        specification="modelarts.vm.cpu.2u",  # For details, see specification.
        instance_count=2,
        envs=envs)                            # (Optional) Configure the environment variable, for example, envs = {"model_name":"mxnet-model-1", "load_epoch":"0"}.
configs.append(service_config1)
configs.append(service_config2)

Method 3: Deploy a batch service transformer.

      
           from modelarts.session import Session
from modelarts.model import Model
from modelarts.config.model_config import TransformerConfig

session = Session()
model_instance = Model(session, model_id='your_model_id')
vpc_id = None                                        # (Optional) ID of the VPC where the batch service instance is deployed. This parameter is left blank by default.
subnet_network_id = None                             # (Optional) Subnet ID. This parameter is left blank by default.
security_group_id = None                             # (Optional) Security group. This parameter is left blank by default.

transformer = model_instance.deploy_transformer(
        service_name="service_transformer_name",
        infer_type="batch",
        vpc_id=vpc_id,
        subnet_network_id=subnet_network_id,
        security_group_id=security_group_id,
        configs=configs                          # transformer configuration parameter. For details, see configs.
)

configs is defined by TransformerConfig in the SDK. The type of configs is list, and the tuple object in the list is TransformerConfig. The code is as follows:

      
           configs = []
mapping_rule = None                               # (Optional) Mapping between input parameters and CSV data
mapping_type= "file"                              # File or CSV
envs = {"model_name":"mxnet-model-1", "load_epoch":"0"}

transformer_config1 = TransformerConfig(
            model_id="model_id",
            specification="modelarts.vm.cpu.2u",   # For details, see specification.
            instance_count=2,
            src_path="/shp-cn4/sdk-demo/",         # OBS path to the input of the batch task, for example, /your_obs_bucket/src_path
            dest_path="/shp-cn4/data-out/",        # OBS path to the output of the batch task, for example, /your_obs_bucket/dest_path
            req_uri="/",
            mapping_type=mapping_type,
            mapping_rule=mapping_rule,
            envs=envs)                             # (Optional) Configure the environment variable, for example, envs = {"model_name":"mxnet-model-1", "load_epoch":"0"}.
configs.append(transformer_config1)

Parameters

**Table 1** Parameters
Parameter	Mandatory	Type	Description
service_id	Yes	String	Service ID, which can be obtained from the real-time service on the ModelArts management console
session	Yes	Object	Session object. For details about the initialization method, see Session Authentication.

**Table 2** Parameters for deploying the predictor and transformer
Parameter	Mandatory	Type	Description
service_name	No	String	Name of a service that consists of 1 to 64 characters and must start with a letter. Only letters, digits, underscores (_), and hyphens (-) are allowed.
description	No	String	Service description, which contains a maximum of 100 characters. By default, this parameter is left blank.
infer_type	No	String	Inference mode. The value can be real-time or batch. The default value is real-time. real-time: real-time service. A model is deployed as a web service and provides real-time test UI and monitoring capabilities. The service keeps running. batch: batch service. A batch service can perform inference on batch data and automatically stops after data processing is completed.
vpc_id	No	String	ID of the VPC to which a real-time service instance is deployed. By default, this parameter is left blank. In this case, ModelArts allocates a dedicated VPC to each user, and users are isolated from each other. To access other service components in the VPC of the service instance, set this parameter to the ID of the corresponding VPC. Once a VPC is configured, it cannot be modified. When vpc_id and cluster_id are configured, only the dedicated cluster parameter takes effect.
subnet_network_id	No	String	ID of a subnet. By default, this parameter is left blank. This parameter is mandatory when vpc_id is configured. Enter the network ID displayed in the subnet details on the VPC management console. A subnet provides dedicated network resources that are isolated from other networks.
security_group_id	No	String	Security group. By default, this parameter is left blank. This parameter is mandatory when vpc_id is configured. A security group is a virtual firewall that provides secure network access control policies for service instances. A security group must contain at least one inbound rule to permit the requests whose protocol is TCP, source address is 0.0.0.0/0, and port number is 8080.
configs	Yes	configs parameters of predictor and transformer	Model running configurations When infer_type is set to batch, only one model can be configured. When infer_type is set to real-time, you can configure multiple models and assign traffic weights based on service requirements. The version numbers of the models must be different.
schedule	No	schedule array	Service scheduling configuration, which can be configured only for real-time services. By default, this parameter is not used. Services run for a long time. For details, see Table 6.
cluster_id	No	String	ID of an old-version dedicated resource pool, which is left blank by default. If this parameter is configured, the service will be deployed in the specified old-version dedicated resource pool.
pool_name	No	String	Name of a new-version dedicated resource pool.

**Table 3** **configs** parameters of **predictor**
Parameter	Mandatory	Type	Description
model_id	Yes	String	Model ID. Obtain the value by calling the API described in Obtaining Models or from the ModelArts management console.
weight	Yes	Integer	Weight of traffic allocated to a model. This parameter is mandatory only when infer_type is set to real-time. The sum of multiple weights must be equal to 100. If multiple model versions are configured in a real-time service and different traffic weights are set, ModelArts continuously accesses the prediction API of the service and forwards prediction requests to the model instances of the corresponding versions based on the weights. { "service_name": "mnist", "description": "mnist service", "infer_type": "real-time", "config": [ { "model_id": "xxxmodel-idxxx", "weight": "70", "specification": "modelarts.vm.cpu.2u", "instance_count": 1, "envs": { "model_name": "mxnet-model-1", "load_epoch": "0" } }, { "model_id": "xxxxxx", "weight": "30", "specification": "modelarts.vm.cpu.2u", "instance_count": 1 } ] }
specification	Yes	String	Resource specifications. The options are modelarts.vm.cpu.2u, modelarts.vm.gpu.p4 (permission required), and modelarts.vm.ai1.a310 (permission required). For the options that require a permission, create a service ticket on Huawei Cloud. Then, ModelArts O&M personnel will add the permissions for you.
instance_count	Yes	Integer	Number of instances deployed in a model. The maximum number of instances is 5. To use more instances, submit a service ticket.
envs	No	Map<String, String>	(Optional) Environment variable key-value pair required for running a model. By default, this parameter is left blank.

**Table 4** **configs** parameters of **transformer**
Parameter	Mandatory	Type	Description
model_id	Yes	String	Model ID
specification	Yes	String	Resource flavor. Currently, modelarts.vm.cpu.2u and modelarts.vm.gpu.p4 are available.
instance_count	Yes	Integer	Number of instances deployed in a model. The value range during the closed beta test is [1, 2].
envs	No	Map<String, String>	(Optional) Environment variable key-value pair required for running a model. By default, this parameter is left blank.
src_path	Yes	String	OBS path of the input data of a batch job
dest_path	Yes	String	OBS path of the output data of a batch job
req_uri	Yes	String	Inference API called in a batch task, that is, the RESTful API exposed in the model image. You must select an API URL from the config.json file of the model for inference. If a built-in inference image of ModelArts is used, the API is displayed as /.
mapping_type	Yes	String	Mapping type of the input data. The value can be file or csv. If you select file, each inference request corresponds to a file in the input data path. When this mode is used, req_uri of a model can have only one input parameter and the type of this parameter is file. If you select csv, each inference request corresponds to a row of data in the CSV file. When this mode is used, the files in the input data path can only be in CSV format and mapping_rule needs to be configured to map the index of each parameter in the inference request body to the CSV file. The following shows how to create a batch service whose mapping_type is set to file: { "service_name": "batchservicetest", "description": "", "infer_type": "batch", "config": [{ "model_id": "598b913a-af3e-41ba-a1b5-bf065320f1e2", "specification": "modelarts.vm.cpu.2u", "instance_count": 1, "src_path": "https://infers-data.obs.xxx.com/xgboosterdata/", "dest_path": "https://infers-data.obs.xxx.com/output/", "req_uri": "/", "mapping_type": "file" }] } The following shows how to create a batch service whose mapping_type is set to csv: { "service_name": "batchservicetest", "description": "", "infer_type": "batch", "config": [{ "model_id": "598b913a-af3e-41ba-a1b5-bf065320f1e2", "specification": "modelarts.vm.cpu.2u", "instance_count": 1, "src_path": "https://infers-data.obs.xxx.com/xgboosterdata/", "dest_path": "https://infers-data.obs.xxx.com/output/", "req_uri": "/", "mapping_type": "csv", "mapping_rule": { "type": "object", "properties": { "data": { "type": "object", "properties": { "req_data": { "type": "array", "items": [{ "type": "object", "properties": { "input5": { "type": "number", "index": 0 }, "input4": { "type": "number", "index": 1 }, "input3": { "type": "number", "index": 2 }, "input2": { "type": "number", "index": 3 }, "input1": { "type": "number", "index": 4 } } }] } } } } } }] }
mapping_rule	No	Map	Mapping between input parameters and CSV data. This parameter is mandatory only when mapping_type is set to csv. The mapping rule is similar to the input parameter definition in the config.json model configuration file. You only need to configure the index parameters under each parameter of the string, number, integer, or boolean type, and the value of this parameter to the values of the index parameters in the CSV file to send an inference request. Use commas (,) to separate multiple pieces of CSV data. The values of the index parameters start from 0. If the value of the index parameter is -1, ignore this parameter. For details, see the sample code of deploying transformer. The format of the inference request body described in mapping_rule is as follows: { "data": { "req_data": [{ "input1": 1, "input2": 2, "input3": 3, "input4": 4, "input5": 5 }] } }

**Table 5** Parameters in the response to the request for deploying **predictor** and **transformer**
Parameter	Mandatory	Type	Description
predictor	Yes	Predictor object	Predictor object. Its attributes include all functions described in this chapter.

**Table 6** **schedule** parameters
Parameter	Mandatory	Type	Description
op_type	Yes	String	Scheduling type. Currently, only the value stop is supported.
time_unit	Yes	String	Scheduling time unit. The options are as follows: DAYS HOURS MINUTES
duration	Yes	Integer	Value that maps to the time unit. For example, if the task stops after two hours, set time_unit to HOURS and duration to 2.

Example of deploying a real-time predictor instance in the handwritten digit recognition project implemented by MXNet:

        
             from modelarts.session import Session
from modelarts.model import Model
from modelarts.config.model_config import ServiceConfig, TransformerConfig

model_instance = Model(session, model_id = "you_model_id")
configs = []
config1 = ServiceConfig(model_id="you_model_id", 
                        weight="100", 
                        instance_count=1, 
                        specification="modelarts.vm.cpu.2u",
                        envs={"input_data_name":"images",
                              "input_data_shape":"0,1,28,28",
                              "output_data_shape":"0,10"})
configs.append(config1)
predictor = model_instance.deploy_predictor(service_name="DigitRecognition", configs=configs)

Example of deploying a transformer instance (batch processing) in a handwritten digit recognition project implemented by MXNet:

        
             from modelarts.session import Session
from modelarts.model import Model
from modelarts.config.model_config import ServiceConfig, TransformerConfig

model_instance = Model(session, model_id = "your_model_id") 
configs = []
config1 = TransformerConfig(model_id="your_model_id", 
                            specification="modelarts.vm.cpu.2u", 
                            instance_count=1, 
                            envs={"input_data_name":"images","input_data_shape":"0,1,28,28","output_data_shape":"0,10"},
                            src_path="/w0403/testdigitrecognition/inferimages/",
                            dest_path="/w0403/testdigitrecognition/" ,
                            req_uri = "/",
                            mapping_type = "file")
configs.append(config1)
predictor = model_instance.deploy_transformer(service_name="DigitRecognition", infer_type="batch", configs=configs)