Lora Qwen-MoE adapter_model.pt merge to base model weight

[iqiancheng]

model:
  _component_: torchtune.models.qwen3.lora_qwen3_moe_30b_a3b_instruct
  lora_attn_modules: ['q_proj', 'k_proj', 'v_proj', 'output_proj', 'w1', 'w2', 'w3', 'output']
  apply_lora_to_mlp: True
  apply_lora_to_output: True

The LoRA training layer extends from the attention layer ['q_proj', 'v_proj', 'output_proj'] to the entire linear layer ['q_proj', 'k_proj', 'v_proj', 'output_proj', 'w1', 'w2', 'w3', 'output'], how to merge adapter_model.pt to base model?

[iqiancheng]

Hi everyone,

I've implemented a script to merge the LoRA adapter weights from adapter_model.pt into the base model, covering all linear layers: ['q_proj', 'k_proj', 'v_proj', 'output_proj', 'w1', 'w2', 'w3', 'output']. I've attached the script to this issue in case it can help others with similar needs.

If you have any suggestions or run into any issues, please feel free to let me know!

qwen3_lora_merge_to_base_hf.py