Notifications

Overview

In large-scale AI training scenarios, training jobs typically run for hours or even days and consume multi-node GPU/NPU cluster resources. When a training job completes or encounters an exception, you need to be promptly notified of the status change to ensure rapid response and handling.

ModelArts supports enabling training job exception notifications by subscribing to Simple Message Notification (SMN), with options to send alerts via voice calls, SMS, and email. When a job completes or fails, the system automatically sends subscription notifications to you, ensuring you are instantly aware of job status changes, can promptly handle exceptions, and can effectively prevent resource waste and time losses caused by prolonged job execution failures.

Prerequisites

ModelArts implements message notification by subscribing to SMN topics. Therefore, before configuring event notifications when creating a training job, you must first complete the creation of topics and subscriptions in the SMN console.

Configuring Event Notifications

ModelArts supports configuring message notifications for model training and reinforcement learning jobs. For specific configuration details, see the documentation for creating the corresponding jobs. The key configurations are as follows:

1. Log in to the ModelArts console.
2. In the navigation pane, choose Model Build > Model Training or Model Build > Reinforcement Learning. The model training or reinforcement learning workspace is displayed.
3. Click Create Training Job or Create in the upper right corner. On the job creation page that is displayed, enable event notifications and configure the notification content. The notification content configuration items are as follows:
   • Topic: Select the notification topic. You must complete the creation of topics and subscriptions in the SMN console in advance.
   • Event: Set the events for which notifications are required. For details on currently supported notification events, see Notification Event Description.
   Figure 1 Configuring event notifications
   
4. After the configuration is complete, create the training job. During the training process, if a configured event is triggered, the system will send notifications according to the subscribed notification method and configuration content.

Notification Event Description

Table 1 shows the event types supported for model training configurations.