Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Video Llava HF fix #713

Merged
merged 34 commits into from
Jan 9, 2025
Merged
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
d2c7606
VILA added
Jul 25, 2024
9a0e0ba
Merge branch 'main' into main
junming-yang Jul 25, 2024
58b2f24
Update README.md
junming-yang Jul 25, 2024
96783ef
fix merge conflict
amitbcp Jul 25, 2024
004614b
resolve config merge conflict
amitbcp Jul 25, 2024
d3149dc
Fix error on Idefics for longer prompt
amitbcp Jul 25, 2024
ee7de8a
Merge branch 'open-compass:main' into main
amitbcp Jul 27, 2024
30523a4
Fix naming convention to make consistent with Idefics2 and better rea…
amitbcp Jul 27, 2024
595eb5f
Merge branch 'main' of https://github.com/amitbcp/VLMEvalKit into main
amitbcp Jul 27, 2024
3695566
Merge branch 'open-compass:main' into main
amitbcp Jul 27, 2024
072f20c
update config for idefics
amitbcp Jul 27, 2024
63561bb
Make LLava consistent as well
amitbcp Jul 27, 2024
2a8486e
Merge branch 'open-compass:main' into main
amitbcp Jul 29, 2024
2ca4d1d
Add VILA 1.5 3B
amitbcp Jul 29, 2024
da9eedd
Add VILA 1.5 3B
amitbcp Jul 29, 2024
28e33dc
fix naming convention to be similar to the HF models
amitbcp Jul 29, 2024
0c0cee5
update main from open-compass
amitbcp Jul 30, 2024
7f18c91
Multi-Turn added for Phi3-Vision and tested with MMDU
amitbcp Jul 30, 2024
e2a15bc
Merge branch 'open-compass:main' into main
amitbcp Aug 3, 2024
cd433af
Merge branch 'open-compass:main' into main
amitbcp Aug 6, 2024
6bc24fc
Merge branch 'open-compass:main' into main
amitbcp Aug 9, 2024
baaea5e
Add multi turn for Intern VL
amitbcp Aug 11, 2024
61f1df0
fix formatting
amitbcp Aug 11, 2024
015d7d3
Merge branch 'open-compass:main' into main
amitbcp Aug 20, 2024
5540840
Merge branch 'open-compass:main' into main
amitbcp Aug 20, 2024
f2d1b96
Merge branch 'open-compass:main' into main
amitbcp Aug 26, 2024
91818e8
Merge branch 'open-compass:main' into main
amitbcp Sep 26, 2024
da94d01
Merge branch 'open-compass:main' into main
amitbcp Oct 25, 2024
97993f9
Merge branch 'open-compass:main' into main
amitbcp Oct 30, 2024
de4dd0c
Merge branch 'open-compass:main' into main
amitbcp Nov 11, 2024
b96a00e
Merge branch 'open-compass:main' into main
amitbcp Jan 6, 2025
f0b7e2a
defined self.nframe as used in the generate_inner
amitbcp Jan 6, 2025
29773dd
add raise xception message in video llava hf
FangXinyu-0913 Jan 9, 2025
4438b6c
Merge branch 'main' into amitbcp/main
FangXinyu-0913 Jan 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions vlmeval/vlm/video_llm/video_llava.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,13 @@ def __init__(self, model_path='LanguageBind/Video-LLaVA-7B-hf', **kwargs):
self.model.eval().cuda()
self.processor = VideoLlavaProcessor.from_pretrained(model_path)
self.kwargs = kwargs
self.nframe = 8
torch.cuda.empty_cache()

def generate_inner(self, message, dataset=None):
import av
if self.nframes != 8:
raise Exception(f'Video-LLaVA only supported 8 frames to generate, you now set frame numbers to {self.nframes}') # noqa
question, video = self.message_to_promptvideo(message)

container = av.open(video)
Expand Down
Loading