FAQs About Training

Q1: Why Does Running the Sample Script Copied from the Documentation Trigger a syntax error near unexpected token `$'{\r''? Error?

A: When you create a file by copying sample scripts from the documentation, the line endings of the Windows system might differ from those of Linux, causing line ending mismatch issues. You can perform format conversion inside the Linux system by running dos2unix xxx.sh, or by opening the file with vim and executing the following two commands:

:set ff=unix
:wq

Q2: Are the Script Parameters Provided in This Document Universally Applicable to All Models? What Should I Keep in Mind When Switching Models?

A: No. When using different models or conducting different types of training, you must refer to the specific script examples tailored to each model from MindSpeed-LLM/docs/quick_start.md · Ascend/MindSpeed-LLM - AtomGit | GitCode.

However, the overall workflow remains the same. You will execute the following scripts in sequence and modify their parameters based on the instructions in this document to start training:

• Weight conversion script (HF to MG format): ckpt_convert_XXX_hf2mcore.sh
• Data processing script: data_convert_XXX_instruction.sh (or data_convert_XXX_pretrain.sh for pre-training tasks)
• Training script: Select tune for fine-tuning, pretrain for pre-training, full for full-parameter fine-tuning, and LoRA for Low-Rank Adaptation.
• Weight conversion script (MG to HF format): ckpt_convert_XXX_mcore2hf.sh
• Main training task script: run_distributed_task.sh must also be adjusted accordingly based on the modifications made above.

Q3: Can I Execute the Model Weight Conversion, Data Format Conversion, or Training Task Scripts Individually Instead of Using the Main Training Startup Script?

A: This is not recommended. Beyond handling cluster communication, the main training task script (run_distributed_task.sh) also manages critical operations such as OBS path configurations and model saving. Furthermore, configuring environment variables within the main script is more standardized and convenient. Using the main startup script is fully supported and recommended even when training on a single instance.

Q4: If the Training Task Fails, but the Weight Conversion and Data Formatting Have Already Executed Successfully and Saved to Storage, Can I Skip Converting Them Again When I Restart the Training?

A: Yes. As long as the parameters and paths remain unchanged, you can simply comment out the weight conversion script (ckpt_convert_XXX_hf2mcore.sh) and the data formatting script (data_convert_XXX_instruction.sh) within the main training script (run_distributed_task.sh). Then, fill in the corresponding model paths, parallelism parameters, and directory paths in run_distributed_task.sh to resume or start training directly.

Q5: Where Is MindSpeed-LLM Installed? How Can I Find This Path If I Want to Make Modifications or Add Print Statements?

A: MindSpeed-LLM is installed under /home/ma-user/MA_Turbo/src/open_source/MindSpeed-LLM, and other dependencies are located under /home/ma-user/MA_Turbo/src/open_source.

Inside the MindSpeed-LLM directory, you will find the exact same branch content and structure as the official repository. When modifying or calling custom MindSpeed-LLM components and files (such as the convert_ckpt.py script), ensure your paths are correct, or explicitly use the absolute path: /home/ma-user/MA_Turbo/src/open_source/MindSpeed-LLM/convert_ckpt.py.

Q6: What Format Are the Weights Saved in After Training Is Complete? Why Does the Number of Files Differ from the Original Hugging Face (HF) Model Files?

A: Immediately after training concludes, the generated weights are in Megatron-Core (mcore) format, where weights are sharded across different GPUs to achieve maximum distributed training efficiency. After being processed by the mg2hf conversion script, these are converted into standard Hugging Face (HF) format weights, which is the universally supported format for mainstream inference engines (such as vLLM).

It is completely normal for the number of converted .safetensors files to differ from the original model due to differences in the configured sharding strategies. Although the file count changes, the model parameters remain fully intact under the mapping of the model.safetensors.index.json index file, allowing the model to be loaded and used normally.

Before using the model, generate a new model.safetensors.index.json based on the weights converted by mg2hf. This prevents issues where the index file still references the old, pre-trained model weight shards.

The v1 version has more examples and offers better stability; it is recommended to use v1 for weight conversion. If you prefer to use v2, ensure that both hf2mg and mg2hf utilize the v2 pipeline. You can review the documentation above to understand the differences between the two.

The weight conversion process for LoRA training differs significantly. Read the introduction in the v1 documentation to modify your training and weight conversion scripts accordingly.

Sample script to generate a new model.safetensors.index.json:

import os
import json
from safetensors import safe_open
# === Modify this to your actual path ===
models_dir = "$TRAINED_HF_MODELS"
# ===========================
def generate_index(dir_path):
    print(f"Scanning directory: {dir_path}")
    files = sorted([f for f in os.listdir(dir_path) if f.endswith(".safetensors")])
    
    if not files:
        print("Error: No .safetensors files found in the directory!")
        return
    weight_map = {}
    total_size = 0
    print(f"Found {len(files)} weight files. Rebuilding index...")
    
    for file_name in files:
        file_path = os.path.join(dir_path, file_name)
        try:
            # Open file to read the internal tensor keys
            with safe_open(file_path, framework="pt") as f:
                keys = f.keys()
                # Register all keys inside this file to the map
                for k in keys:
                    weight_map[k] = file_name
                    
            # Get file size (optional, used for metadata)
            total_size += os.path.getsize(file_path)
            print(f"Processed: {file_name} ({len(keys)} parameters)")
        except Exception as e:
            print(f"Failed to read file {file_name}: {e}")
            return
    # Construct the final index dictionary
    index_dict = {
        "metadata": {
            "total_size": total_size
        },
        "weight_map": weight_map
    }
    # Save the new index.json
    save_path = os.path.join(dir_path, "model.safetensors.index.json")
    with open(save_path, "w") as f:
        json.dump(index_dict, f, indent=2)
    
    print("-" * 30)
    print(f"Success! New index file generated at: {save_path}")

if __name__ == "__main__":
    generate_index(models_dir)