Executing a Training Job

This section describes how to configure and execute a training job on the ModelArts training platform.

Prerequisites

You have configured permissions on the ModelArts console. For details, see Configuring Agency Authorization for ModelArts with One Click. If you have not configured permissions, a message will be displayed on the top of the training job page, prompting you to configure permissions.
The model weights, training data, data preprocessing script, and training script have been uploaded to an OBS bucket, for example, obs://verl/verl-a2.

Step 1: Configuring Basic Settings

Click Create Training Job. Set the basic parameters of the training job according to the following table.

**Table 1** Basic configuration
Parameter	Mandatory	Description	Example
Training Mode	Yes	The training mode can be Fine-Tuning or Custom Job. In this example, select Custom Job.	Custom Job
Name	Yes	(Mandatory) Name of the training job. The system automatically generates a name, which you can then rename.	Automatically generated
Description (Optional)	No	Job description, which helps you learn about the job information in the training job list.	VeRL training

Step 2: Defining Training Configuration

Set the key parameters of the training job according to Table 2.

**Table 2** Training configuration
Parameter	Mandatory	Description	Example
Select Image	Yes	Select the container image required for training. In this example, select a preset image. The image environment required for VeRL framework training is preset on the platform and can be directly used.	Preset Images
Image address	Yes	Image path. The VeRL image contains the software required for training, such as the CANN package, driver, PyTorch, and VeRL framework code package.	VeRL
Boot Command	Yes	Command for starting the training image. The cd operation for switching to the local code directory in the container is mandatory. You need to upload the training job startup script and associated script files to your OBS bucket in advance. Qwen3-8B requires the following files: run_train_8b.sh, run_qwen3-8b_npu_ma.sh, gsm8k.py Qwen2.5-VL-32B-Instruct requires the following files: run_train_32b.sh, run_qwen2_5_vl_32b_npu_ma.sh, geometry3k.py Qwen3-30B-A3B requires the following files: run_train_30b.sh, run_qwen3-30b_a3b_npu_ma.sh, gsm8k.py	Qwen3-8B cd /home/ma-user/work/verl-a2 bash /home/ma-user/work/verl-a2/run_train_8b.sh Qwen2.5-VL-32B-Instruct cd /home/ma-user/work/verl-a2 bash /home/ma-user/work/verl-a2/run_train_32b.sh Qwen3-30B-A3B cd /home/ma-user/work/verl-a2 bash /home/ma-user/work/verl-a2/run_train_30b.sh
Code Directory	Yes	If the code is stored in OBS, select the code directory in the OBS bucket path.	obs://verl/verl-a2
Local Code Directory	Yes	Specify the local directory of the training container. When training starts, the system automatically downloads the code directory to this directory. Change the value to /home/ma-user/work, which is the same as the code directory set in the training script.	/home/ma-user/work
Environment Variable	No	The environment variables are already set in the training script, so you do not need to set them here.	N/A

Step 3: Configuring Resources

Set the resource configuration parameters of the training job according to Table 3.

**Table 3** Resource configuration
Parameter	Mandatory	Description	Example
Source of resources	Yes	Public and dedicated resource pools are available for you to select. In this example, use a dedicated resource pool.	Dedicated resource pool
Resource Pool	Yes	Click Select Resource Pool. In the dialog box displayed on the right, select a physical resource pool with an 8 x Snt9b2 specification. The resource pool must be created in advance. Hover over View in the Resource Fragment column to check fragment details and check whether the resource pool meets the training requirements.	Physical 8 x Snt9b2
Specifications	Yes	Select an 8 x Snt9b2 specification.	8 x Snt9b2
Compute Nodes	Yes	Qwen3-8B model training requires 8-PU resources per node. Set this parameter to 1. Qwen2.5-VL-32b-Instruct model training requires 16-PU resources. Set this parameter to 2.	Qwen3-8B: 1 Qwen2.5-VL-32b-Instruct: 2 Qwen3-30b-A3B: 2
Storage Mounting	Yes	Click Add Extended Storage (OBS) to store training artifacts.	Extended Storage (OBS)
		Directory: Select the training output path in the OBS bucket.	obs://verl/verl-a2
		Mount Path: Output path of the training artifacts in the training container.	/output_dir
		readonly: The mounted OBS path can only be read and cannot be written. The training artifacts need to be written to OBS. Do not select this option.	Do not select it.
Job Scheduling Priority	No	Scheduling priority of the training job.	Retain the default settings.
Preemption	No	When enabled, jobs that allow preemption may be terminated and re-queued if resource pool capacity is insufficient. To avoid losing training progress, configure resumable training before enabling this function.	Do not select it.

Step 4: Configuring HA

Set the HA configuration parameters of the training job according to Table 4.

**Table 4** HA configuration
Parameter	Mandatory	Description	Example
Maximum Restarts	No	The system marks the job as failed if it continues to fail after reaching the maximum restart limit. The default value is 3.	Retain the default settings.
Unconditional Auto Restart	No	As long as a training exception is detected, ModelArts unconditionally restarts the training job. To prevent invalid restarts, it supports a maximum of three consecutive unconditional restarts.	Select it.
Restart Upon Suspension	No	ModelArts continuously monitors job processes to detect suspension and optimize resource usage. When this feature is enabled, suspended jobs can be automatically restarted at the process level. To prevent unnecessary restarts, ModelArts limits consecutive restarts to three.	Select it.

Step 5: Managing Access Configuration

In this example, retain the default settings for parameters related to online debugging and remote SSH in the access configuration.

Step 6: Enabling Observability

In this example, retain the default settings for Interconnect Metrics with AOM in the observability configuration.

Step 7: Adjusting Additional Configurations

Set the additional configuration parameters of the training job according to Table 5.

**Table 5** Additional configuration
Parameter	Mandatory	Description	Example
Persistent Log Saving	Yes	Logs will be deleted after 30 days. If you enable this function, logs will be saved to a specified OBS path. You can also download all logs on the job details page to a local path.	Select it. obs://verl/verl-a2/logs/
Job Visibility	No	By default, the job is visible to all users in the same workspace. Creator: The created training job is invisible to other users.	Workspace
Auto Stop	No	If this function is enabled, the job tracks its runtime. It stops automatically if the runtime surpasses the set time limit.	Do not select it.
Event Notification	No	If enabled, you will receive SMS or email notifications when specific events occur (such as job status changes or suspected suspension). These notifications may incur minor charges.	Do not select it.
Tags	No	To use the same tag for multiple cloud resources, set the tag in TMS.	Do not select it.

At the bottom of the Create Training Job page, click Submit to create the training job.

A training job runs for a period of time. You can go to the training job list or training details page to view the basic information about the training job.

After the training is complete, view the training logs and performance by referring to Viewing the Training Output. The training output is stored in the OBS bucket obs://verl/verl-a2.

For more information about ModelArts training, see Model Training.

Parent topic: Training the Qwen3 & Qwen2.5-VL Model Adapted to the verl Framework on ModelArts

Previous topic: Preparing the Training Scripts

Next topic: Viewing the Training Output