Include Stable Diffusion into AM-RADIO ?

[javiabellan]

I think Stable Diffusion would be a great teacher for AM-RADIO aswell because of the following papers:

	paper	Poster
Diffusion Hyperfeatures	https://arxiv.org/abs/2305.14334	https://diffusion-hyperfeatures.github.io
SD Complements DINO	https://arxiv.org/abs/2305.15347	https://sd-complements-dino.github.io
LDM correspondences	https://arxiv.org/pdf/2305.15581	https://ubc-vision.github.io/LDM_correspondences
Diffusion Features (DIFT)	https://arxiv.org/pdf/2306.03881	https://diffusionfeatures.github.io
Telling Left from Right	https://arxiv.org/abs/2311.17034	https://telling-left-from-right.github.io
Probe 3D	https://arxiv.org/abs/2404.08636	https://github.com/mbanani/probe3d

BTW: SD-based methods achieve SOTA on semantic-correspondence

There are many other models to consider (like MiDaS for Depth estimation). But I think SD bring somethig new because of its UNET backbone and its generative loss. Screenshot from the Probe3D paper:

Update: Code for obtain feats from SD2.1

Links from the DIFT repo:

• https://github.com/Tsingularity/dift/blob/main/src/models/dift_sd.py
• https://colab.research.google.com/drive/1km6MGafhAvbPOouD3oo64aUXgLlWM6L

[mranzinger]

Wow, thanks for putting this together. We'll look into it. In general, we agree that adding a generative diffusion model could make for a great addition.