[CVPR2025] Decouple-Then-Merge: Finetune Diffusion Models as Multi-Task Learning

🔥 News

• [2025/02/27] DeMe has been accepted to CVPR 2025! 🤗🤗
• [2025/05/23] Code has been released! 🤗🤗

📝 TODO

• Training code
• Inference code
• Project page
• Journal version

🛠️ Getting Started

To setup the environment of DeMe, follow the installation instructions below.

1. Clone the github repo

git clone https://github.com/MqLeet/DeMe.git

2. Create conda environment

cd DeMe
conda env create -f environment.yml
conda activate DeMe

3. Prepare the pretrained model (An example on LSUN-Church)

We use an official DDPM model offered by Google in Huggingface, named ddpm-ema-church-256. You can download the pretrained DDPM model at ddpm-ema-church-256/ for the following experiments.

The used pretrained models can be found here:

Model	Link
ddpm-cifar10-32	https://huggingface.co/google/ddpm-cifar10-32
ddpm-ema-church-256	https://huggingface.co/google/ddpm-ema-church-256
ddpm-ema-bedroom-256	https://huggingface.co/google/ddpm-ema-bedroom-256
stable-diffusion-v1-4	https://huggingface.co/CompVis/stable-diffusion-v1-4

4. Dataset preparation (An example on LSUN-Church)

For convenience, we use LSUN-Church from Huggingface without downloading data by using the load_dataset function offered by the datasets library. Additionally, you can also download the dataset yourself.

For evaluation, you can download the .npz files at Google Drive

📘 Usage (Experiments on LSUN-Church as an example)

Decouple: Finetune DDPM at 4 Different Timestep Ranges

Finetune the pretrained DDPM and save finetuned model

sh lsun_church_deme.sh

Merge: Merge Finetuned Diffusion Models into One Unified Model

Compute task vectors and merge finetuned diffusion models into one diffusion model

python merge.py

For the convenience of the users, we have provided the merged model weights on the LSUN-Church, which can be downloaded at Google Drive. You can download the merged model weights to path 'merged_models/diffusion_pytorch_model.bin' and continue to evaluate the FID in the following steps.

Inference:

To accelerate sampling process, we use multiple GPUs to sample

sh merge_run_ddp_single.sh

Evaluation:

We offer a church_fid.npz file here to evaluate the generated images quality, use the command line to compute FID

python -m pytorch_fid "ddim_samples/merge/reproduce_deme" "church_fid.npz"

to reproduce the results demonstrated in paper.

✒️ Citation

If you find our repo useful for your research, please consider citing our paper:

   @InProceedings{Ma_2025_CVPR,
   author    = {Ma, Qianli and Ning, Xuefei and Liu, Dongrui and Niu, Li and Zhang, Linfeng},
   title     = {Decouple-Then-Merge: Finetune Diffusion Models as Multi-Task Learning},
   booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
   month     = {June},
   year      = {2025},
   pages     = {23281-23291}
   }

❤️ Acknowledgement

Our code is built upon diffusers and task vectors. We also refer to the model soups and dnn mode connectivity. Thanks to the contributors for their great work!