Visualizing the Training Job Process

Description

When training machine learning models, you must track the status of your training jobs. Traditional methods often fail to show all necessary details, making it hard to follow progress accurately. With this feature, you can check the real-time status of a training job across multiple dimensions like job scheduling, environment setup, and execution progress. The visual interface allows you to check the entire lifecycle of a training job, helping you monitor progress, adjust parameters, and improve both efficiency and user experience.

Constraints

The main phases of the job process remain constant, while the subphases can vary. For example, if a training job lacks input data, the "input is downloading" sub-phase under "Environment Preparing" will be absent. Similarly, if the "job initializing environment is checking" event is missing, its corresponding sub-phase within the "Job Running" phase will also be omitted.

Checking Training Phases

On the top of the training job details page, the training job module of ModelArts Standard displays the job process details. There are four main phases: Job Scheduling, Environment Preparing, Job Running, and Job End.

You can check the subphases of each main phase.

• Job Scheduling logs all job activities, including successful or failed creations, scheduled jobs, and their exact timestamps.
• Environment Preparing documents essential steps for preparing the environment. It includes starting the setup, checking initial conditions, downloading training code, and completing the process, along with timestamps.
• Job Running tracks job details, including start and end times for each training step.
• Job End captures essential details like the date and time of the job's outcome.

Click Records in the upper right corner of the job process to see all its activity logs. This allows you to track details like repeated preemptions, rescheduling, or restarts during the job's execution.

Figure 1 Job process