Viewing the Training Output
The training output, such as preprocessed training data, checkpoints, logs, and model weights, is stored in the obs://<bucket_name>/verl-a2 bucket.
The following shows the training result structure:
|——{OBS bucket} # Custom bucket name, for example, verl
|──{verl-a2} # Custom OBS folder name, for example, verl-a2
|──{ckpts} # Custom OBS folder for storing checkpoints, for example, ckpts
|──{GRPO-Qwen3} # Custom OBS folder name, which is the project name, for example, GRPO-Qwen3
|──{GRPO-Qwen3-8B-npu} # Custom OBS folder, which is the experiment name, for example, GRPO-Qwen3-8B-npu.
|──latest_checkpointed_iteration.txt # Iteration for saving the latest checkpoint
|──global_step_{number} # Training weight folder for step {number}
|──run_qwen3-8b_npu_ma.log # Training log
|──run_qwen2_5_vl_32b_npu_ma.log # Training log
|──run_qwen3_30b_a3b_npu_ma.log # Training log Viewing Logs
Search for obs://<bucket_name>/verl-a2/run_qwen3_8b_npu_ma.log, obs://<bucket_name>/verl-a2/run_qwen2_5_vl_32b_npu_ma.log, or obs://<bucket_name>/verl-a2/run_qwen3_30b_a3b_npu_ma.log in your OBS bucket.
Checking Performance
Training performance is mainly checked based on two metrics in the training logs: throughput and convergence.
- Throughput (tokens/s/p): The performance is calculated based on the verl_grpo-{af_model_name}-<Sequence length-Device type-Timestamp>-run_log-N1-WS8.txt file in the path specified by ${output_dir}. The perf/throughput field in the training log file indicates the throughput of a single-step calculation. You can use the visualization tool TrainingLogParser to view the average training throughput.
- Score convergence: The critic/score/mean value in the log approaches 1 and stabilizes over time during training iterations. You can also use the visualization tool TrainingLogParser to view the convergence of critic/score/mean.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot