Executing a Training Job

This section describes how to configure and execute a training job on the ModelArts console.

Prerequisites

You have configured permissions on the ModelArts console. For details, see Configuring Agency Authorization for ModelArts with One Click. If you have not configured permissions, a message will be displayed on the top of the training job page, prompting you to configure permissions.
The model weights, training data, data preprocessing script, and training script have been uploaded to an OBS bucket, for example, obs://areal/xxx.

Step 1: Configuring Basic Settings

Click Create Training Job. Set the basic parameters of the training job according to the following table.

**Table 1** Basic configuration
Parameter	Mandatory	Description	Example
Training Mode	Yes	The training mode can be Fine-Tuning or Custom Job. In this example, select Custom Job.	Custom Job
Name	Yes	(Mandatory) Name of the training job. The system automatically generates a name, which you can then rename.	Automatically generated
Description (Optional)	No	Job description, which helps you learn about the job information in the training job list.	AReaL training

Step 2: Defining Training Configuration

Set the key parameters of the training job according to Table 1.

Note: When using OBS, it is recommended that you use either the train.save_steps or train.save_epochs parameter in the boot command and set the other parameter to 0. VeOmni saves folders named global_step_XX. If save_steps and save_epochs have the same value, this can cause duplicate writes in OBS, leading to an I/O error.

For details about parameter settings, see Arguments API Reference — VeOmni v0.1.0 documentation.

**Table 2** Training configuration
Parameter	Mandatory	Description	Example
Select Image	Yes	Select the container image required for training. In this example, select a preset image. The image environment required for VeOmni framework training is preset on the platform and can be directly used.	Preset Images
Image address	Yes	Image path. The VeOmni image contains the software required for training, such as the CANN package, driver, PyTorch, and VeOmni framework code package.	VeOmni
Boot Command	Yes	Boot command of the training image. You need to upload the training job startup script and associated script files to your OBS bucket in advance. After configuring all parameters in the script, run train.sh. This script contains configurations for multi-node communication, fine-tuning tasks, model saving, weight conversion, and checks. In the boot command, run the cd command to go to the Veomni directory.	Multi-node training cd /home/ma-user/VeOmni bash "${input_dir}/train.sh" /home/ma-user/VeOmni/tasks/train_torch.py /home/ma-user/VeOmni/configs/sft/qwen3_sft.yaml \ --model.model_path "${model_dir}/Qwen3-32B" \ --data.train_path "${dataset_dir}/tulu-first2000.parquet" \ --train.data_parallel_mode fsdp2 \ --train.init_device meta \ --train.output_dir "${output_dir}/Qwen3-32B-Base-sft-tulu" \ --train.save_epochs 3 \ --train.num_train_epochs 3 \ --train.ulysses_parallel_size 2 \ --data.max_seq_len 4096 \ --train.global_batch_size 8 The same script is used for single-node training.
Code Directory	Yes	If the code is in OBS, choose the OBS bucket path obs://veomni/veomni-a2/. During training, all files from the OBS path are copied to the training container's directory. Any changes made to this directory are not synced back to the OBS bucket. If the files in the OBS bucket are too big, downloading them to the training container can take a long time. This makes the job start slowly. It is recommended that the script, model folder, and dataset folder of a model training job be mounted to the same OBS bucket path and used only for this training job.	obs://veomni/veomni-a2/
Local Code Directory	Yes	Local directory of the training container. When the training starts, the system downloads the code directory from OBS to this directory and overwrites the existing content. Use a newly created directory. If the code directory in OBS is set to obs://veomni/veomni-a2/ and the local code directory is set to /home/ma-user/modelarts/user-job-dir, the directory structure in the container is as follows: /home/ma-user/modelarts/user-job-dir/veomni-a2/	/home/ma-user/modelarts/user-job-dir
Environment Variable	No	When using a training script, add an output environment variable. For example, set output_dir as the parameter name (customizable). The output path /home/ma-user/modelarts/user-job-dir/veomni/output/veomni-output saves preprocessed data, checkpoints, and training logs to OBS. Set the environment variables for the model directory (model_dir), script directory (input_dir), and dataset directory (dataset_dir). Note: To use an extended OBS path, create a folder with the same name as your OBS storage location in the cloud mount path and then mount this folder. For example, if your OBS storage location is obs://modelarts/Template/veomni/veomni-output/ and your cloud mount path is /home/ma-user/modelarts/user-job-dir/veomni/output, set the environment variable output_dir to /home/ma-user/modelarts/user-job-dir/veomni/output/veomni-output. This stores the output in OBS. To avoid confusion, use different names for the bottom-layer folders in OBS and the cloud mount path.	Parameter: output_dir Value: /home/ma-user/modelarts/user-job-dir/veomni/output/veomni-output

Step 3: Configuring Resources

Set the resource configuration parameters of the training job according to Table 3.

**Table 3** Resource configuration
Parameter	Mandatory	Description	Example
Source of resources	Yes	Public and dedicated resource pools are available for you to select. In this example, use a dedicated resource pool.	Dedicated resource pool
Resource Pool	Yes	Click Select Resource Pool. In the dialog box displayed on the right, select a physical resource pool with an 8 x Snt9b2 specification. The resource pool must be created in advance. Hover over View in the Resource Fragment column to check fragment details and check whether the resource pool meets the training requirements.	Physical 8 x Snt9b2
Specifications	Yes	Select an 8 x Snt9b2 specification.	8 x Snt9b2
Compute Nodes	Yes	Qwen3-8B model training requires two nodes with 16-PU resources. Set this parameter to 2. Qwen3-32B model training requires three nodes with 24-PU resources. Set this parameter to 3.	Qwen3-8B: 2 Qwen3-32B: 3
Storage Mounting	No	Click Add Extended Storage (OBS) to store training artifacts.	Extended Storage (OBS)
		Directory: Select the training output path in the OBS bucket. The path must differ from the OBS code path.	obs://<bucket_name>/ckpt_space
		Mount Path: Output path of the training artifacts in the training container. Example: If obs://<bucket_name>/ckpt_space is mounted to the container, the path is /home/ma-user/ckpts/ckpt_space.	/home/ma-user/ckpts
		readonly: The mounted OBS path can only be read and cannot be written. The training artifacts need to be written to OBS. Do not select this option.	Do not select it.
Job Scheduling Priority	No	Scheduling priority of the training job.	Retain the default settings.
Preemption	No	When enabled, jobs that allow preemption may be terminated and re-queued if resource pool capacity is insufficient. To avoid losing training progress, configure resumable training before enabling this function.	Do not select it.

Step 4: Configuring HA

Set the HA configuration parameters of the training job according to Table 4.

**Table 4** HA configuration
Parameter	Mandatory	Description	Example
Maximum Restarts	No	The system marks the job as failed if it continues to fail after reaching the maximum restart limit. The default value is 3.	Retain the default settings.
Unconditional Auto Restart	No	As long as a training exception is detected, ModelArts unconditionally restarts the training job. To prevent invalid restarts, it supports a maximum of three consecutive unconditional restarts.	Do not select it.
Restart Upon Suspension	No	ModelArts continuously monitors job processes to detect suspension and optimize resource usage. When this feature is enabled, suspended jobs can be automatically restarted at the process level. To prevent unnecessary restarts, ModelArts limits consecutive restarts to three.	Select it.

Step 5: Managing Access Configuration

In this example, retain the default settings for parameters related to online debugging and remote SSH in the access configuration.

Step 6: Enabling Observability

In this example, retain the default settings for Interconnect Metrics with AOM in the observability configuration.

Step 7: Adjusting Additional Configurations

Set the additional configuration parameters of the training job according to Table 5.

**Table 5** Additional configuration
Parameter	Mandatory	Description	Example
Persistent Log Saving	Yes	Logs will be deleted after 30 days. If you enable this function, logs will be saved to a specified OBS path. You can also download all logs on the job details page to a local path. The path can be configured as needed.	Select it.
Job Visibility	No	By default, the job is visible to all users in the same workspace. Creator: The created training job is invisible to other users.	Workspace
Auto Stop	No	If this function is enabled, the job tracks its runtime. It stops automatically if the runtime surpasses the set time limit.	Do not select it.
Event Notification	No	If enabled, you will receive SMS or email notifications when specific events occur (such as job status changes or suspected suspension). These notifications may incur minor charges.	Do not select it.
Tags	No	To use the same tag for multiple cloud resources, set the tag in TMS.	Do not select it.

At the bottom of the Create Training Job page, click Submit to create the training job.

A training job runs for a period of time. You can go to the training job list or training details page to view the basic information about the training job.

After the training is complete, view the training logs and performance by referring to Viewing the Training Output.

For more information about ModelArts training, see Model Training.

Parent topic: Training the Qwen3 Series Models Adapted to the AReaL Framework on ModelArts

Previous topic: Preparing the Training Scripts

Next topic: Viewing the Training Output