About finetuning the instructblip on my own datasets

[VoyageWang]

Hi, thanks for your great work! I wonder what should I do if I want to fine-tune the instructblip via lora on my own dataset. First, what will be the format of my dataset (QA, instruction tuning)? Second, what should I change in the code?

[waitzkin]

Hello, I apologize for the delayed response.

Unfortunately, the question you asked covers the overall implementation of our work, so I think it would be difficult to explain in text. I recommend you look at the "git diff" between our code and the original LAVIS repo to see what changes are needed.

The dataset format can be either a QA dataset or an instruction tuning dataset, and you can implement the appropriate class for each. For the instruction template, you can reference the original InstructBLIP paper.

If you have further questions or need in-depth help, please feel free to email me at [email protected].