How Do I Speed Up Real-Time Prediction?

When using real-time inference services, you can improve prediction performance in the following ways:

• When deploying a real-time inference service, you can choose a resource pool with better performance and more compute to boost prediction performance. The compute of the resource pool directly determines the inference response speed and concurrent processing capacity. Prioritizing high-performance resources unlocks inference potential at the hardware level.
• When deploying a real-time inference service, configure the number of deployment replicas properly.

  If the number of replicas is set to 1, performance depends entirely on a single instance, and reliability is insufficient—individual failures will cause service unavailability. You are advised to configure the number of replicas to be greater than 1. This not only enhances the concurrent handling capacity of the overall service but also ensures high availability and service reliability through multi-replica redundancy.

• Inference performance is closely tied to the model itself and the inference framework used. You can choose an inference engine equipped with pre-configured optimization settings (such as the preset model optimization configurations in TensorRT-LLM or vLLM). Alternatively, you can optimize the model through model compression, quantization, or structural streamlining to reduce inference latency. ModelArts provides model asset management capabilities, making it easy to trace lineage and repeatedly fine-tune models.
• ModelArts provides preset inference service images and model assets that have already undergone framework adaptation and parameter tuning, enabling direct, one-click deployment.