How Do I Promptly Detect Service Container Exceptions During Real-Time Service Deployment or Instance Restart?

Symptom

In cloud service deployment scenarios, users typically rely on storage solutions such as Object Storage Service (OBS), Parallel File System (PFS), and SFS Turbo to store and manage model weight files. During model service deployment or instance restart, the system must download these files from the storage services and mount them to service containers to complete the deployment process. However, in actual operation, if any of the dependency storage services encounter issues, such as network interruptions, service unavailability, or insufficient disk space, the system may fail to detect these exceptions effectively. This can lead to deployment failures or prolonged service unavailability, without timely exception detection, thereby impacting service reliability and user experience.

Configuring LTS Log Alarms

To address this issue, the inference platform integrates with the LTS system. Through LTS log-based alerts, developers can be promptly notified of anomalies during the model weight file reading process, helping ensure smooth service deployment.

1. After the service is interconnected with LTS, access the service details page, and switch to the Logs tab to view logs.
2. Click Log Settings. In the displayed dialog box, click the Alarm Rules tab.
3. Click Create. In the displayed dialog box, configure the alarm rule, set Statistics to By keyword, and analyze the keywords in the error logs recorded when the weight file fails to be read based on the selected image and weight. Enter the keywords in the keyword alarm rule of LTS.

   For details about the parameters, see Configuring Log Alarm Rules.

   To add a notification when an alarm is generated, enable and create alarm notification rules on the LTS console. For details, see Creating a Message Template on the LTS Console.

4. Click OK.

   You can change the alarm settings in the log settings dialog box. For details, see Follow-up Operations on Alarm Rules.

Configuring OBS/SFS Turbo Metric Alarm Rules on Cloud Eye

You can also view OBS/SFS Turbo metrics on Cloud Eye to locate faults. For details, see Creating an Alarm Rule and Notifications. For details about OBS metrics, see OBS Monitoring Metrics.For details about SFS Turbo metrics, see SFS Turbo Metrics.