[model] support for microsoft's Phi-4-mini

[ctx289]

🎯 Brief Introduction

Phi-4-mini-instruct is a lightweight open model built upon synthetic data and filtered publicly available websites - with a focus on high-quality, reasoning dense data. The model belongs to the Phi-4 model family and supports 128K token context length. The model underwent an enhancement process, incorporating both supervised fine-tuning and direct preference optimization to support precise instruction adherence and robust safety measures.

Primary Use Cases

The model is intended for broad multilingual commercial and research use. The model provides uses for general purpose AI systems and applications which require:

• Memory/compute constrained environments
• Latency bound scenarios
• Strong reasoning (especially math and logic).

The model is designed to accelerate research on language and multimodal models, for use as a building block for generative AI powered features.

Model

• github
• huggingface

Usage for Training

Create a new file examples/train_lora/phi4_mini_lora_sft.yaml with the following content:

model_name_or_path: microsoft/Phi-4-mini-instruct

trust_remote_code: true

### method

stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all

### dataset

dataset: identity,alpaca_en_demo
template: phi4_mini

cutoff_len: 2048
max_samples: 1000
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output

output_dir: saves/phi-4-mini-3.8B/lora/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train

per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

### eval

# eval_dataset: alpaca_en_demo

# val_size: 0.1

# per_device_eval_batch_size: 1

# eval_strategy: steps

# eval_steps: 500

used

1 gpu

DISABLE_VERSION_CHECK=1 llamafactory-cli train examples/train_lora/phi4_mini_lora_sft.yaml
 1 node

DISABLE_VERSION_CHECK=1 FORCE_TORCHRUN=1 llamafactory-cli train examples/train_lora/phi4_mini_lora_sft.yaml

result

template: phi

template: phi4_mini

We conducted a comparison of the loss metrics between the phi template and phi4_mini template. The results indicate that the ​phi4_mini ​template is better suited for the training process of the current ​phi4_mini ​model (demonstrating better initial loss convergence and format compatibility with the target model.

[gemini-code-assist]

Summary of Changes

Hello @ctx289, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates Microsoft's Phi-4-mini-instruct model, a compact yet powerful language model known for its strong reasoning capabilities and efficiency, into the system. The changes involve defining a dedicated chat template to ensure proper interaction formatting and registering the model with its official sources, making it readily available for users seeking high-quality, resource-efficient AI solutions.

Highlights

• New Model Support: Added comprehensive support for Microsoft's Phi-4-mini-instruct model, a lightweight yet powerful language model.
• Custom Chat Template: Introduced a dedicated chat template named phi4_mini to ensure proper formatting and interaction with the Phi-4-mini model.
• Model Registration: Registered the Phi-4-3.8B-instruct model, linking it to its official Hugging Face and ModelScope download sources and associating it with the new phi4_mini template.
• Documentation Update: Updated both English and Chinese README files to reflect the inclusion of Phi-4-mini in the list of supported models.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request successfully adds support for Microsoft's Phi-4-mini model, including updates to the documentation, registration of the new model's template, and integration into the model group constants. The changes are well-implemented and follow the existing patterns in the codebase.

[hiyouga]

LGTM