Executing a Training Job

This section describes how to configure and execute a training job on the ModelArts console.

Prerequisites

You have configured permissions on the ModelArts console. For details, see Configuring Agency Authorization for ModelArts with One Click. If you have not configured permissions, a message will be displayed on the top of the training job page, prompting you to configure permissions.
Upload the model weight, training data, and training scripts to an OBS bucket, for example, obs://mindspeed/mindspeed-a2.

Step 1: Configuring Basic Settings

Click Create Training Job. Set the basic parameters of the training job according to the following table.

**Table 1** Basic configuration
Parameter	Mandatory	Description	Example
Training Mode	Yes	The training mode can be Fine-Tuning or Custom Job. In this example, select Custom Job.	Custom Job
Name	Yes	(Mandatory) Name of the training job. The system automatically generates a name, which you can then rename.	Automatically generated
Description (Optional)	No	Job description, which helps you learn about the job information in the training job list.	MindSpeed-LLM training

Step 2: Defining Training Configuration

Set the key parameters of the training job according to Table 2.

**Table 2** Training configuration
Parameter	Mandatory	Description	Example
Select Image	Yes	Select the container image required for training. In this example, select a preset image. The image environment required for MindSpeed-LLM framework training is preset on the platform and can be directly used.	Preset Images
Image address	Yes	Image path. The MindSpeed-LLM image contains the software required for training, such as the CANN package, driver, PyTorch, and MindSpeed-LLM framework code package.	MindSpeed-LLM
Boot Command	Yes	Boot command of the training image. You need to upload the training job startup script and associated script files to your OBS bucket in advance. Once all parameters in the scripts are configured, simply execute run_distributed_task.sh. This script encompasses configurations for weight conversion, data processing, fine-tuning tasks, as well as model saving and checks. For the boot command, you must first cd into the local code directory inside the container where your main training startup script is stored.	Multi-node training cd /home/ma-user/MA_Turbo/src/open_source/MindSpeed-LLM/code bash run_distributed_task.sh The same script is used for single-node training.
Code Directory	Yes	If the code is stored in OBS, select the OBS bucket path obs://mindspeed-llm/mindspeed-llm-a2/code_directory. During training, all files from the OBS path are copied to the training container's directory. Any changes made to this directory are not synced back to the OBS bucket. If the files in the OBS bucket are too big, downloading them to the training container can take a long time. This makes the job start slowly. It is recommended that the script, model folder, and dataset folder of a model training job be mounted to the same OBS bucket path and used only for this training job.	obs://mindspeed-llm/mindspeed-llm-a2/
Local Code Directory	Yes	Local directory of the training container. When the training starts, the system downloads the code directory from OBS to this directory and overwrites the existing content. Use a newly created directory. MindSpeed-LLM and its dependency installation directory are in /home/ma-user/MA_Turbo/src/open_source/MindSpeed-LLM. Create a directory as the local code directory, for example, /home/ma-user/MA_Turbo/src/open_source/MindSpeed-LLM/code. If the OBS code directory is obs://mindspeed-llm/input/, the directory structure in the container is as follows: /home/ma-user/MA_Turbo/src/open_source/MindSpeed-LLM/code/input/	/home/ma-user/MA_Turbo/src/open_source/MindSpeed-LLM/code
Environment Variable	No	When using a training script, add an output environment variable. For example, set output_dir as the parameter name (customizable). The output path /home/ma-user/modelarts/outputs/output_dir_0 saves preprocessed data, checkpoints, and training logs to OBS. Similarly, configure environment variables for the model path for training (e.g., model_path) and the dataset path to be processed (e.g., dataset_path). Note: To use an extended OBS path, create a folder with the same name as your OBS storage location in the cloud mount path and then mount this folder. For example, if your selected extended storage OBS path is /test/mindspeed-llm/output_dir_0 and your cloud mount path is set to /home/ma-user/modelarts/outputs, setting the environment variable output_dir to /home/ma-user/modelarts/outputs/output_dir_0 will ensure all outputs are correctly saved directly to OBS.	Parameter: output_dir Value: /home/ma-user/modelarts/outputs/output_dir_0

Step 3: Configuring Resources

Set the resource configuration parameters of the training job according to Table 3.

**Table 3** Resource configuration
Parameter	Mandatory	Description	Example
Source of resources	Yes	In this example, use a dedicated resource pool. If you select a public resource pool, training process files cannot be saved.	Dedicated resource pool
Resource Pool	Yes	Click Select Resource Pool. In the dialog box displayed on the right, select a physical resource pool with an 8 x Snt9b2 specification. The resource pool must be created in advance. Hover over View in the Resource Fragment column to check fragment details and check whether the resource pool meets the training requirements.	Physical 8 x Snt9b2
Specifications	Yes	Select an 8 x Snt9b2 specification.	8 x Snt9b2
Compute Nodes	Yes	Qwen3-8B model training requires 8-PU resources per node. Set this parameter to 1. Qwen3-30B-A3B model training requires 16-PU resources. Set this parameter to 2. Qwen3-32B model training requires 16-PU resources. Set this parameter to 2.	Qwen3-8B: 1 Qwen3-30B-A3B: 2 Qwen3-32B: 2
Storage Mounting	Yes	Click Add Extended Storage (OBS) to store training artifacts.	Extended Storage (OBS)
		Directory: Select the training output path in the OBS bucket.	obs://mindspeed-llm/mindspeed-llm-a2/{output}
		Mount Path: Output path of the training artifacts in the training container.	/home/ma-user/modelarts/outputs/output_dir_0
		readonly: The mounted OBS path can only be read and cannot be written. The training artifacts need to be written to OBS. Do not select this option.	Do not select it.
Job Scheduling Priority	No	Scheduling priority of the training job.	Retain the default settings.
Preemption	No	When enabled, jobs that allow preemption may be terminated and re-queued if resource pool capacity is insufficient. To avoid losing training progress, configure resumable training before enabling this function.	Do not select it.

Step 4: Configuring HA

Set the HA configuration parameters of the training job according to Table 4.

**Table 4** HA configuration
Parameter	Mandatory	Description	Example
Maximum Restarts	No	The system marks the job as failed if it continues to fail after reaching the maximum restart limit. The default value is 3.	Retain the default settings.
Unconditional Auto Restart	No	As long as a training exception is detected, ModelArts unconditionally restarts the training job. To prevent invalid restarts, it supports a maximum of three consecutive unconditional restarts.	Select it.
Restart Upon Suspension	No	ModelArts continuously monitors job processes to detect suspension and optimize resource usage. When this feature is enabled, suspended jobs can be automatically restarted at the process level. To prevent unnecessary restarts, ModelArts limits consecutive restarts to three.	Select it.

Step 5: Managing Access Configuration

In this example, retain the default settings for parameters related to online debugging and remote SSH in the access configuration.

Step 6: Enabling Observability

In this example, retain the default settings for Interconnect Metrics with AOM in the observability configuration.

Step 7: Adjusting Additional Configurations

Set the additional configuration parameters of the training job according to Table 5.

**Table 5** Additional configuration
Parameter	Mandatory	Description	Example
Persistent Log Saving	Yes	Logs will be deleted after 30 days. If you enable this function, logs will be saved to a specified OBS path. You can also download all logs on the job details page to a local path.	Select it.
Job Visibility	No	By default, the job is visible to all users in the same workspace. Creator: The created training job is invisible to other users.	Workspace
Auto Stop	No	If this function is enabled, the job tracks its runtime. It stops automatically if the runtime surpasses the set time limit.	Do not select it.
Event Notification	No	If enabled, you will receive SMS or email notifications when specific events occur (such as job status changes or suspected suspension). These notifications may incur minor charges.	Do not select it.
Tags	No	To use the same tag for multiple cloud resources, set the tag in TMS.	Do not select it.

At the bottom of the Create Training Job page, click Submit to create the training job.

A training job runs for a period of time. You can go to the training job list or training details page to view the basic information about the training job.

After the training is complete, view the training logs and performance by referring to Viewing the Training Output. The training output is stored in the OBS bucket.

For more information about ModelArts training, see Model Training.

Parent topic: Training the Qwen3 Series Models Adapted to the MindSpeed-LLM Framework on ModelArts

Previous topic: Preparing the Training Scripts

Next topic: Viewing the Training Output