This project defines the pipeline to train VLM models based on opensource implementation of CLIP style models.
We recommend using a conda environment.
conda create -n vlm_training python=3.11
conda activate vlm_training
pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu126
pip install -r requirments.txtThis pipeline consists of three main stages:
- Configuration generation
- Individual train/evaluation scripts generation
- Aggregate job submission in array jobs for training and evaluation
Generate configurations using NePS:
python configs_generator.py \
--hpo-optimizer scan_space_final_biased_alpha_0_5 \
--seed 94 \
--workspace_dir "/path/to/workspace"
This creates sampled configurations under:
/path/to/workspace/results/optimizer=scan_space_final_biased_alpha_0_5/seed=94
Iterate over the sampled configurations and prepare training commands:
python configs_iterator.py \
--root_dir "results/optimizer=scan_space_final_biased_alpha_0_5/seed=94" \ # keep relative
--seed 94 \
--workspace_dir "/path/to/workspace" \
--cluster meta \
--partition alldlc_gpu-rtx3080 \
--wandb_project_name vlm_training \
--command_dir "/path/to/command/dir" \
--bias_scale_sampling \
--alpha 0.5
Generate cluster ready job scripts:
python array_jobs_generator.py \
--seed 94 \
--root_dir "results/optimizer=scan_space_final_biased_alpha_0_5/seed=94" \
--wandb_project_name vlm_training \
--cluster meta \
--command_dir "/path/to/command/dir" \
--workspace_dir "/path/to/workspace" \
--partition alldlc_gpu-rtx3080
All scripts are created under:
workspace_dir/root_dir
- train_configX.sh for single configuration training
- train_p{partition}_bs{batch_size}_g{nr_gpus}_n{nr_nodes}.sh for grouped jobs
- eval_configX.sh for single configuration evaluation
- array_job.sh for batched evaluation
Submit jobs to the cluster in the following order:
- Training
- Upstream evaluation
After all jobs have finished, aggregate the results:
python neps_status.py \
--root_dir "results/optimizer=scan_space_final_biased_alpha_0_5/seed=94" \
--start_from 1 \
--end_at 1001
This generates a CSV file containing the aggregated results.
- Training must complete before evaluation starts
- Designed for cluster execution (e.g. Slurm based systems). Check
slurm.pyfor more details. - Downstream evaluation is performed using: https://github.com/mlfoundations/datacomp. See their code on how to evaluate each model.
A cluster script used to download the data: download_train_data.sh
Full information on the trained configurations is found here
Run the following script from the main source-tree to recreate the plottings:
bash train/figures.shThis project builds upon existing open-source software. We gratefully acknowledge the contributions of the original authors.
Our implementation partially reuses and adapts code from:
- OpenClip (https://github.com/mlfoundations/open_clip)
- Clip benchmark (https://github.com/LAION-AI/CLIP_benchmark)
- NePs(https://github.com/automl/neps)
All third-party components remain under their respective licenses.
Full license texts are provided in THIRD_PARTY_LICENSES.md.