Updated on 2026-09-03 GMT+08:00

Viewing the Training Output

The training output, such as preprocessed training data, checkpoints, logs, and model weights, is stored in the obs://<bucket_name>/verl-a2 bucket.

The following shows the training result structure:

|——{OBS bucket}                                                   # Custom bucket name, for example, verl
   |──{verl-a2}                                              # Custom OBS folder name, for example, verl-a2
       |──{ckpts}                                            # Custom OBS folder for storing checkpoints, for example, ckpts
            |──{GRPO-Qwen3}                                  # Custom OBS folder name, which is the project name, for example, GRPO-Qwen3
                 |──{GRPO-Qwen3-8B-npu}                      # Custom OBS folder, which is the experiment name, for example, GRPO-Qwen3-8B-npu.
                      |──latest_checkpointed_iteration.txt   # Iteration for saving the latest checkpoint
                      |──global_step_{number}                # Training weight folder for step {number}
       |──run_qwen3-8b_npu_ma.log                            # Training log
       |──run_qwen2_5_vl_32b_npu_ma.log                      # Training log
       |──run_qwen3_30b_a3b_npu_ma.log                       # Training log

Viewing Logs

Search for obs://<bucket_name>/verl-a2/run_qwen3_8b_npu_ma.log, obs://<bucket_name>/verl-a2/run_qwen2_5_vl_32b_npu_ma.log, or obs://<bucket_name>/verl-a2/run_qwen3_30b_a3b_npu_ma.log in your OBS bucket.

Checking Performance

Training performance is mainly checked based on two metrics in the training logs: throughput and convergence.

  1. Throughput (tokens/s/p): The performance is calculated based on the verl_grpo-{af_model_name}-<Sequence length-Device type-Timestamp>-run_log-N1-WS8.txt file in the path specified by ${output_dir}. The perf/throughput field in the training log file indicates the throughput of a single-step calculation. You can use the visualization tool TrainingLogParser to view the average training throughput.
  2. Score convergence: The critic/score/mean value in the log approaches 1 and stabilizes over time during training iterations. You can also use the visualization tool TrainingLogParser to view the convergence of critic/score/mean.