Configuring Service Information

Overview

The inference service information provides a central portal to ensure a permanent and stable call URL even as your models iterate. You can easily define authentication methods, network access policies, and global traffic switches here.

Billing

Service information is not billed.

Procedure

1. Log in to the ModelArts console. In the navigation pane, choose Model Inference > Real-Time Inference.
2. In the real-time service list, click Deploy.
3. On the displayed page, configure service information.
   Figure 1 Basic information
   

   Table 1 Basic information

   Parameter	Description	Example
   Service Name	Name used to identify and manage the real-time service. Enter a name as prompted. Only letters, digits, hyphens (-), and underscores (_) are supported. Max length: 128 characters	service-e16d The name is automatically generated by the system.
   Description (Optional)	Brief description for a real-time service.	/
   Service Type	Real-time service type. You can evaluate the performance of your text generation service after deployment. Supported types include text generation, image understanding, video generation, image generation, reranking, vector, embedding, and other.	Text generation
   Service Call Mode	Real-time services support synchronous and asynchronous calls. Synchronous call: Once your request is submitted, the connection remains active while waiting for the server to return the processing result, until the final response is received or a timeout occurs. This mode is ideal for short-duration inference tasks that require real-time results. The call mode cannot be modified after the service is created. Asynchronous call: Once your request is submitted, the service will immediately acknowledge receipt and process it on the backend. You can retrieve inference results later via polling. This mode is ideal for scenarios with long inference times, such as AIGC or video processing, where long-connection timeouts may cause request failures or load imbalances across instances. The call mode cannot be modified after the service is created. For details, see Asynchronous Inference.	Synchronous
   Max Tasks per Service	When the service call mode is set to asynchronous, you need to set the maximum number of tasks that can be created for a single service. The value ranges from 0 to 10,000. Tasks in the succeeded, failed, created_failed, start_failed, upgrade_failed, timeout, or deleted state do not occupy the task quota. You can obtain the task status by calling the API for viewing task details.	/
   Auto Stop	Auto-stop timer. Default: 1 hour; maximum: 24 hours. When auto stop is enabled, the system tracks how long the service runs. It will shut down the service if the runtime goes beyond the set limit. After the real-time service is deployed, you can choose Model Inference > Real-Time Inference on the console. Choose More > Configure Auto Stop and reconfigure the auto stop time.	Select it.

   Figure 2 Network settings

   

   Table 2 Network settings

   Parameter	Description	Example
   Service Access Method	Default: Leverages ModelArts capabilities to provide limited public network access via the platform's default prediction URL. Supports authentication on the platform.	Retain the default settings.
   Service Protocol	Protocol used by the inference API and model process API provided by the real-time service. Set this parameter based on the defined inference API. Available protocols: HTTPS, HTTP, WSS, and WS. Using HTTP or WS plaintext can leave you vulnerable to threats like data leakage, tampering, traffic hijacking, and phishing attacks. Proceed with caution.	HTTPS
   Authentication Mode	API KEY, IAM Token, and None are supported. To use an API key for authentication, create one and bind it to the service on the API Key Authorization Management page after deploying the real-time service. Selecting None allows the client to call the API directly, which compromises security. This option is only suitable for temporary testing or internal network calls. Avoid using this mode on external networks. For details, see Accessing a Real-time Service via Different Authentication Methods.	API KEY
   ELB (Optional)	The ELB instance's private IP address or EIP serves as the entry point for inference service requests. User requests are sent to the inference link gateway via the ELB. When you enable this function, the system sets up a listener, a backend server group, and backend servers on the specified ELB. If the VPC associated with the ELB accessed by you is connecting for the first time, a VPC endpoint will be automatically created under this VPC to bridge the network between this VPC and the real-time service. The IP address of this VPC endpoint will serve as the backend server for the ELB. The selected ELB specifications must meet the following requirements: instance type: dedicated; load type: application, or network + application; network configuration: The IP as a Backend option must be selected. For details about how to access an inference service through ELB, see Accessing a Real-Time Service Through ELB.	Retain the default settings.
   More Settings	Public Network Access: Specifies whether to allow external network access to the real-time service. This function is enabled by default. If this function is enabled, external networks can access the service. View the API URL on the service details page. If this function is disabled, real-time services cannot be accessed over the Internet. Auto-approved Private Network Connection: Specifies whether private network connection to real-time services requires approval. When this function is enabled, private network connection requests from third-party users will be approved automatically. If this function is disabled, approval is required. For details about how to access a service through a private network, see Accessing a Real-Time Service Through a Private Network. Access Control: Once selected, you can specify a whitelist or blacklist for access control. Whitelist: Only user IP addresses from the CIDR blocks configured here are allowed access. You can add up to ten regular expressions. Blacklist: Only user IP addresses from the CIDR blocks configured here not allowed access. You can add up to ten regular expressions.	Retain the default settings.

   Figure 3 High availability settings
   

   Table 3 High availability settings

   Parameter	Description	Example
   Rate Limiting Policy	A rate limiting policy consists of a Policy Type and a Rate Limiting Mode. Duplicate policies cannot be added. Multiple policies will be combined and applied as a rule set. For details, see Table 4.	Retain the default settings.
   Request Timeout (s)	The maximum time to wait for the system to return the first token result after a request is sent. The value must range from 1 to 1,200. If no response is received within this period, the request is automatically terminated. In streaming transmission scenarios, the timeout interval is refreshed each time a response to a request is received. However, the end-to-end response timeout interval of the system is fixed at 3,600 seconds.	Retain the default settings.

   Table 4 Rate limiting policy parameters

   Parameter	Sub-Parameter	Description	Example
   Policy Type	Request Size Limit	The maximum size allowed for data contained within a single request. Request size range: 1 to 50. The unit is MB.	Retain the default settings.
   	Requests Per Second	The maximum number of times the service can be accessed per second. Unit: times. The value must range from 1 to 10,000.	Retain the default settings.
   Rate Limiting Mode	Service-level throttling	Sets a fixed total rate limit for the service. New requests are rejected immediately once the threshold is exceeded.	Retain the default settings.
   	Instance-level throttling	The actual service rate is dynamically adjusted based on the number of deployed replicas. Actual service rate = Deployment replica threshold x Number of running deployment replicas.	Retain the default settings.

   Figure 4 Advanced settings
   

   Table 5 Advanced settings

   Parameter	Description
   Ingest Logs to LTS	If Ingest Logs to LTS is not enabled on the service deployment page, only real-time service logs are displayed. When this function is enabled, logs are uploaded to Log Tank Service (LTS) for management. You are billed on a pay-per-use basis. For details, see LTS Pricing Details. LTS automatically creates log groups and streams and caches logs for seven days by default. You can search for and analyze runtime logs on LTS. Log search: Search logs using specific keywords or phrases. Narrow your results by selecting a specific time range to find events and issues during that period. Statistical charts: After sending logs to LTS, use SQL analysis syntax to find important log data and view the results as statistical charts. Log analysis: Before searching for analyzing logs, set up structured data and indexing for them. Real-time logs: Once you connect your real-time service logs to LTS, they will be sent every minute. You can view these updates from the Real-Time Logs tab, where you can also easily search and analyze the data. For details, see Log Tank Service.
   Container stdout	After you select Ingest Logs to LTS, Container stdout is selected by default and cannot be modified. Collect all container standard output and report it to Log Tank Service (LTS). The log retention period follows the setting of the corresponding log group, which is 30 days by default: For dedicated resource pools, the LTS log groups and log streams are those created and selected during the installation of the resource pool log collection plugin. For public resource pools, the system automatically creates LTS log groups and log streams. The naming conventions are: Log group: Modelarts-Infer-Log-Group-{NUM}; log stream: Inf-Stdout-{serviceName}-{First 8 characters of converted serviceId}.
   Kubernetes events	After you select Ingest Logs to LTS, you can manually select Kubernetes events. Once enabled, Kubernetes events (pod events) will be collected and reported to LTS. By default, logs are retained for 7 days. You can view pod events in the event list on the inference service details page. For details, see Viewing Events of a Real-Time Service. If this option is not selected, you can only view pod events of the last hour in the event list on the inference service details page.
   Collect container file logs	After you select Ingest Logs to LTS, you can manually select Collect container file logs. Collect container log files of the current service and report them to Log Tank Service (LTS). The log retention period follows the setting of the corresponding log group, which is 30 days by default. You need to specify the directory and file name where log files are stored inside the container. Configuration instructions: A log directory of /var/logs/* and a file name of *.log represents all files with a .log extension across all directories under /var/logs. A log directory of /var/logs/app_* and a file name of *.log represents all files with a .log extension in any directory matching the app_* pattern under /var/logs. Duplicate path configurations will only take effect once. Common paths: Device log: /home/ma-user/ascend/log/run/device-*/ Run plog: /home/ma-user/ascend/log/run/plog/ Debug plog: /home/ma-user/ascend/log/debug/plog/ System log: /var/logs/ Exception log: /home/ma-user/ascend/log/exceptionDump/
   Tags	ModelArts can work with Tag Management Service (TMS). When creating resource-consuming tasks in ModelArts, for example, training jobs, add tags for these tasks so that ModelArts can use tags to manage resources by group. You can select a predefined TMS tag from the tag drop-down list or customize a tag. Predefined tags are available to all service resources that support tags. Custom tags are available only to the service resources of the user who has created the tags. For details about how to use tags, see Using TMS Tags to Manage Resources by Group.

4. After configuring the service information, click Create Service Only. The system automatically returns to the real-time inference service list page.

   You can also click Next to continue the service deployment configuration. For details, see Follow-Up Operations.

Follow-Up Operations

In the real-time inference service list, choose More > Add Deployment in the Operation column and configure the inference deployment information. For details, see Deploying a Real-Time Inference Service Using a Single Node.

Figure 5 Add Deployment