small model backbone

[gc625-kodifly]

Hi, I've modified the code to load a DinoV2 small model, but i realized that the embed_dim of the vit_small model

def vit_small(patch_size=16, **kwargs):
    model = DinoVisionTransformer(
        patch_size=patch_size,
        embed_dim=384,
        depth=12,
        num_heads=6,
        mlp_ratio=4,
        block_fn=partial(Block, attn_class=MemEffAttention),
        **kwargs,
    )
    return model

is 384, which is causing dimension mismatch problem with the RoMa ckpt provided. which assumes that the embed_dim is 1024. e.g.

 proj16 = nn.Sequential(nn.Conv2d(1024, 512, 1, 1), nn.BatchNorm2d(512))

can you provide the weights for RoMa-s? As the model takes ~6GB vram even after applying the change from kde to approx_kde in #23, so being able to use a small model would help a lot

[Parskatt]

Sorry don't have it. I'll try training one (also with vit base).

[gc625-kodifly]

please do :) that would be really helpful. Looking forward to the results!

[Dawars]

@Parskatt Could you also train one using FeatUp? Would it be possible to get higher resolution outputs?

https://github.com/mhamilton723/FeatUp

[Parskatt]

@Dawars I'm not convinced that FeatUp is useful. Would be glad to be proven wrong, but not something I'll spend time on currently.