Add compositionality benchmarks

[mehdidc]

• CREPE: https://openaccess.thecvf.com/content/CVPR2023/papers/Ma_CREPE_Can_Vision-Language_Foundation_Models_Reason_Compositionally_CVPR_2023_paper.pdf, https://github.com/RAIVNLab/CREPE
• ARO https://github.com/mertyg/vision-language-models-are-bows
• others?

[HarmanDotpy]

listing some others:
SugarCREPE: https://github.com/RAIVNLab/sugar-crepe/tree/main
VL-Checklist: https://github.com/om-ai-lab/VL-CheckList
Winoground: https://huggingface.co/datasets/facebook/winoground
maybe SVO (verb understanding): https://github.com/deepmind/svo_probes
maybe VALSE: https://github.com/Heidelberg-NLP/VALSE

[mehdidc]

Thank you very much @HarmanDotpy !

[mehdidc]

Ok SugarCREPE is there, will look at the rest

[mehdidc]

Another one to consider: https://github.com/arijitray1993/COLA, https://cs-people.bu.edu/array/research/cola/

[mehdidc]

Also:

• https://arxiv.org/abs/2310.19785 (What's "up" with vision-language models? Investigating their struggle with spatial reasoning)
• https://arxiv.org/abs/2310.08577 (Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models)
• https://github.com/yonatanbitton/wysiwyr (What You See is What You Read? Improving Text-Image Alignment Evaluation)

[vishaal27]

Add MMVP-VLM (winoground-style dataset): https://github.com/tsb0601/MMVP?tab=readme-ov-file
https://huggingface.co/datasets/MMVP/MMVP_VLM/

[vishaal27]

Add https://github.com/Top34051/colorswap

[escorciav]

@mehdidc sugarcrepe is there & runs, but there is an issue with the run all functionality. Refer to #128

[mehdidc]

New datasets are being added in https://github.com/LAION-AI/CLIP_benchmark/tree/compositionality_tasks, which will have benchmarks we studied in https://arxiv.org/abs/2506.08227v1

[escorciav]

Looking forward to it!

Perhaps of your interest FS-CLIP project also evaluated over a suite of visio-linguo compositional datasets/benchmarks. Unfortunately, the code of the latter isn't available.