Adding a fine-tuned model to use?

[enkie358]

Hello,

I have been fine-tuning a Moondream2 model with my own data, and would like to use this model with taggui. I have been replacing the downloaded Moondream2 model.safetensor file within the cache, but I was wondering if there was a more refined way to do this, particularly being able to maintain the native Moondream2 model without having to delete it and have taggui re-download it.

Thank you!

[jhc13]

Which version of moondream2 did you finetune from?

[jhc13]

If it was finetuned from the same version that TagGUI downloads (the 2024-03-13 version) or an older version, it will be compatible with TagGUI.

Then you can:

1. Specify a models directory in the settings (this can be any directory)
2. Create another directory inside the models directory and include "moondream2" in its name
3. Place your finetuned model files inside the directory
4. Restart TagGUI and load the model from the top of the models list

If it was finetuned from a newer version, it will probably not be compatible with TagGUI because there were some changes to the model code.

[enkie358]

Hello!

On retraining the moondream2 model to the 2024-03-13 version, and following the above direction and selecting Start Auto-captioning, the following error is displayed:

Traceback (most recent call last):
File "auto_captioning\captioning_thread.py", line 532, in run
File "auto_captioning\captioning_thread.py", line 528, in run
File "auto_captioning\captioning_thread.py", line 415, in run_captioning
File "auto_captioning\captioning_thread.py", line 210, in load_processor_and_model
File "transformers\models\auto\tokenization_auto.py", line 909, in from_pretrained
ValueError
:
Unrecognized configuration class <class 'transformers_modules.vikhyatk.moondream2.fa8398d264205ac3890b62e97d3c588268ed9ec4.configuration_moondream.MoondreamConfig'> to build an AutoTokenizer.
Model type should be one of AlbertConfig, AlignConfig, BarkConfig, BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BlipConfig, Blip2Config, BloomConfig, BridgeTowerConfig, BrosConfig, CamembertConfig, CanineConfig, ChineseCLIPConfig, ClapConfig, CLIPConfig, CLIPSegConfig, ClvpConfig, LlamaConfig, CodeGenConfig, CohereConfig, ConvBertConfig, CpmAntConfig, CTRLConfig, Data2VecAudioConfig, Data2VecTextConfig, DbrxConfig, DebertaConfig, DebertaV2Config, DistilBertConfig, DPRConfig, ElectraConfig, ErnieConfig, ErnieMConfig, EsmConfig, FalconConfig, FastSpeech2ConformerConfig, FlaubertConfig, FNetConfig, FSMTConfig, FunnelConfig, GemmaConfig, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, GPTSanJapaneseConfig, GroundingDinoConfig, GroupViTConfig, HubertConfig, IBertConfig, IdeficsConfig, Idefics2Config, InstructBlipConfig, JambaConfig, JetMoeConfig, JukeboxConfig, Kosmos2Config, LayoutLMConfig, LayoutLMv2Config, LayoutLMv3Config, LEDConfig, LiltConfig, LlamaConfig, LlavaConfig, LlavaNextConfig, LongformerConfig, LongT5Config, LukeConfig, LxmertConfig, M2M100Config, MambaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MgpstrConfig, MistralConfig, MixtralConfig, MobileBertConfig, MPNetConfig, MptConfig, MraConfig, MT5Config, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, NezhaConfig, NllbMoeConfig, NystromformerConfig, OlmoConfig, OneFormerConfig, OpenAIGPTConfig, OPTConfig, Owlv2Config, OwlViTConfig, PaliGemmaConfig, PegasusConfig, PegasusXConfig, PerceiverConfig, PersimmonConfig, PhiConfig, Phi3Config, Pix2StructConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, Qwen2Config, Qwen2MoeConfig, RagConfig, RealmConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RetriBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, SeamlessM4TConfig, SeamlessM4Tv2Config, SiglipConfig, Speech2TextConfig, Speech2Text2Config, SpeechT5Config, SplinterConfig, SqueezeBertConfig, StableLmConfig, Starcoder2Config, SwitchTransformersConfig, T5Config, TapasConfig, TransfoXLConfig, TvpConfig, UdopConfig, UMT5Config, VideoLlavaConfig, ViltConfig, VipLlavaConfig, VisualBertConfig, VitsConfig, Wav2Vec2Config, Wav2Vec2BertConfig, Wav2Vec2ConformerConfig, WhisperConfig, XCLIPConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, YosoConfig.

[jhc13]

I'm sorry I missed this. What did you name the model? It should include moondream2 in its name.

[BenDes21]

Hello,

I have been fine-tuning a Moondream2 model with my own data, and would like to use this model with taggui. I have been replacing the downloaded Moondream2 model.safetensor file within the cache, but I was wondering if there was a more refined way to do this, particularly being able to maintain the native Moondream2 model without having to delete it and have taggui re-download it.

Thank you!

Hi there! possible to have a guide to how to fine tune a model ?

[enkie358]

Hello!

My team adapted the following workbook for finetuning moondream2:

https://github.com/vikhyat/moondream/blob/main/notebooks/Finetuning.ipynb

Cheers!

[BenDes21]

Hello!

My team adapted the following workbook for finetuning moondream2:

https://github.com/vikhyat/moondream/blob/main/notebooks/Finetuning.ipynb

Cheers!

Thanks @enkie358 , do you think you can explain the few steps for non-tech people like me ? Basically I have to run the script with a dataset of images with manual captions right ?

Let's said I want " remove " the censorship / add explicits words in the model, do I have to write manually the captions of nude pics with explicit vocabulary then train the img+captions with the script ? Thanks a lot

[enkie358]

Yes, you will have to re-write parts of the script code to point to your source directory and ground truth captions. You will likely want to direct your questions to the github of the model you want to finetune (e.g. https://github.com/vikhyat/moondream), as they will be able to help you further, as this is likely outside the scope of the taggui maintainer. I watched some youtube tutorials and also worked in a small team to work on the code, Alternatively there are active projects like https://github.com/modelscope/swift that are working on a gui. I have not yet installed this and still am looking at it, and most of the support appears to be in Chinese, but this may be more useful. Also, implementing a finetuned model (at least a moondream one) is not without its own challenges.

[BenDes21]

Yes, you will have to re-write parts of the script code to point to your source directory and ground truth captions. You will likely want to direct your questions to the github of the model you want to finetune (e.g. https://github.com/vikhyat/moondream), as they will be able to help you further, as this is likely outside the scope of the taggui maintainer. I watched some youtube tutorials and also worked in a small team to work on the code, Alternatively there are active projects like https://github.com/modelscope/swift that are working on a gui. I have not yet installed this and still am looking at it, and most of the support appears to be in Chinese, but this may be more useful. Also, implementing a finetuned model (at least a moondream one) is not without its own challenges.

thanks a lot for your useful answer, awesome Im gonna check Swift

[generrosity]

I don't think you have a wiki page here, but it would be quite cool to see someone (!) create a page or dedicated discussion with steps

ie - minimal steps with the default model to add a face, and working in the app. This would be a great baseline to lead people into the world of finetuning and models. Then, extended examples would be where additional dependencies and more complex situations come up, option changes, or refactoring, and how individuals solved them.

I say 'someone' as jhc13 might not be as interested, but @enkie358 and @BenDes21 may have some some minimalist steps to do this along the way.