How to load a fine-tuned model with HF trainer script and save it into local

[abdelkareemkobo]

model_reloaded = timm.create_model('hf_hub:nateraw/resnet18-random', pretrained=True)
I tried to load the dinov2 with the following
model = timm.create_model('hf_hub:facebook/dinov2-base',pretrained=True)
I get the following error

--------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[34], [line 2](vscode-notebook-cell:?execution_count=34&line=2)
      [1](vscode-notebook-cell:?execution_count=34&line=1) # Load black box model for explanations
----> [2](vscode-notebook-cell:?execution_count=34&line=2) model_reloaded = timm.create_model('hf_hub:facebook/dinov2-base', pretrained=True)
      [4](vscode-notebook-cell:?execution_count=34&line=4) #! model = timm.create_model('vit_base_patch16_224', pretrained=True)
      [5](vscode-notebook-cell:?execution_count=34&line=5) model = timm.create_model('vit_base_patch14_dinov2', pretrained=True)

File ~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_factory.py:102, in create_model(model_name, pretrained, pretrained_cfg, pretrained_cfg_overlay, checkpoint_path, scriptable, exportable, no_jit, **kwargs)
     [99](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_factory.py:99) assert not pretrained_cfg, 'pretrained_cfg should not be set when sourcing model from Hugging Face Hub.'
    [100](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_factory.py:100) # For model names specified in the form `hf-hub:path/architecture_name@revision`,
    [101](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_factory.py:101) # load model weights + pretrained_cfg from Hugging Face hub.
--> [102](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_factory.py:102) pretrained_cfg, model_name, model_args = load_model_config_from_hf(model_name)
    [103](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_factory.py:103) if model_args:
    [104](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_factory.py:104)     for k, v in model_args.items():

File ~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_hub.py:145, in load_model_config_from_hf(model_id)
    [143](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_hub.py:143) pretrained_cfg = hf_config
    [144](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_hub.py:144) hf_config = {}
--> [145](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_hub.py:145) hf_config['architecture'] = pretrained_cfg.pop('architecture')
    [146](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_hub.py:146) hf_config['num_features'] = pretrained_cfg.pop('num_features', None)
    [147](https://file+.vscode-resource.vscode-cdn.net/home/kareem/hacking/research/AI_Love/Dinov2_for_cancer/lung_cancer_Iq/vit_XAI/CausalX-ViT/ViT_CX/~/mambaforge/envs/XAI/lib/python3.10/site-packages/timm/models/_hub.py:147) if 'labels' in pretrained_cfg:  # deprecated name for 'label_names'

KeyError: 'architecture

[rwightman]

'facebook/dinov2-base' is a transformers model not a timm model, timm options that work with timm.create_model are:

• https://huggingface.co/timm/vit_base_patch14_dinov2.lvd142m
• https://huggingface.co/timm/vit_base_patch14_reg4_dinov2.lvd142m (with registers)

Whene there are architectures that exist across both timm and transformers like this, the weights are almost always in different formats with different key names, sometimes different structure in terms of nesting and qkv vs q, k, v being separate or fused layers, etc. There's always a way to remap them but you have to have a bit of experience to do that easily (or give Claude the two model impl and ask for a weight translation, it's usually pretty close).

[abdelkareemkobo]

thanks i understand this after sometime,
can you help me in somethine very quick.
Is it possible to use vit_base_patch14_dinov2.lvd142m
with the following code and whay should i change for this.

[rwightman]

@abdelkareemkobo there's a lot of code there, but looks like some of it is oriented to timm vit so should be possible to get it working. I also just threw up an attn viz space for timm models, still a WIP with improvements needed, but the basics are working for normal vits. https://huggingface.co/spaces/timm/timmAttentionViz

[abdelkareemkobo]

I think your gradio app is helpful,
I will try to use with my fine tuned medical Dinov2.
I tried with there code but get vision transformer has no reprol! Some errors i don't understand.
You are doing a great work though