Viewing the Training Output

Logs and Weights

The output is saved in the path you set. If you choose the output variable ${output_dir}, you can see the output in the path configured in OBS.

Viewing Logs

You can view the logs in your OBS bucket or on the terminal of the task page.

Checking Performance

Training performance is mainly checked based on two metrics in the training logs: throughput and convergence.

You can run npu-smi info in the shell to view the real-time usage.

Alternatively, you can view the overall throughput and processing time in the training logs.

The logs below show each step takes about 4,500 ms and processes 4,096 tokens. Parameters such as the number of samples used in each step and the learning rate are also displayed in the logs.

Each worker's tasks can vary, and their log prints might differ. Not all workers show the training details for each step. If the primary worker does not display these details, switch to another worker.

Viewing the Saved Results

After the training is complete, the data is transferred, and the task exits, you can view the saved results in the corresponding OBS path.

ckpt: Weights saved in mg format after training.

finetune_dataset: The processed dataset ready for fine-tuning.

model_weights: Model weights converted from hf to mg format.

trained_hf_models: The model weight folder containing weights converted back from mg to hf format based on the checkpoints. The script automatically copies the model configuration files (e.g., config.json) and tokenizer files (e.g., tokenizer.model, vocab.json) into this folder.