Skip to content

Issues: EvolvingLMMs-Lab/lmms-eval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

191 Open 81 Closed
191 Open 81 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Results differ when running multiple tasks
#569 opened Mar 4, 2025 by pumetu
[BUG] NameError: name 'random' is not defined in qbench/utils.py causes failure in qbench task.
#565 opened Mar 3, 2025 by TongXin-CASIA
port
#564 opened Mar 2, 2025 by Marcher-lam
Unrecognized AriaConfig
#557 opened Feb 26, 2025 by trinhvg
The version does not match the llava version.
#556 opened Feb 26, 2025 by Code24Man
Get N/A when evaluating lmms-lab/llava-onevision-qwen2-7b-ov on mathvista_testmini, docvqa_test, and infovqa_test
#553 opened Feb 24, 2025 by viyjy
5
Bug in qwen2-vl ?
#552 opened Feb 23, 2025 by LiJunscs
4
Support of VideoChat-Flash
#551 opened Feb 22, 2025 by hello-bluedog
new
#549 opened Feb 21, 2025 by Marcher-lam
How to use LLM as judge?
#548 opened Feb 21, 2025 by whc688
1
[Request] Support for MileBench
#547 opened Feb 21, 2025 by commmet-ahn
1
gen_kwargs['until'] defaults to "\n\n"
#543 opened Feb 19, 2025 by nanocm
Use --use_cache but the responses are not saved
#542 opened Feb 17, 2025 by XXXCARREY
Any plan for supporting ""LVBench: An Extreme Long Video Understanding Benchmark""
#539 opened Feb 14, 2025 by RexGPT
MMMU-test doesn't generate submission files
#538 opened Feb 13, 2025 by zjuAJW
AttributeError: 'dict' object has no attribute 'to_dict'
#537 opened Feb 12, 2025 by xiGHK
2
MLVU aggregate function is incorrect
#533 opened Feb 11, 2025 by SCZwangxiao
video mme
#532 opened Feb 11, 2025 by Violettttee
1
TypeError: 'JpegImageFile' object is not iterable
#512 opened Jan 27, 2025 by lmx760581375
2
Qwen2-VL,IndexError: index 0 is out of bounds for dimension 0 with size 0
#506 opened Jan 19, 2025 by cxli2333
long time waiting after postprocessing
#504 opened Jan 17, 2025 by yiyexy
1
reproduce the LLaVA-NeXT-Video 7B scores
#503 opened Jan 16, 2025 by junho-jo
1
[OOM] CUDA out of memory error during evaluating llava-hf and llama-vision models
#500 opened Jan 14, 2025 by commmet-ahn
2
ValueError: BuilderConfig 'testdev_balanced_instructions' not found. Available: ['default']
#499 opened Jan 14, 2025 by lmx760581375
2
load_video gets 0 frame for mp4 file
#495 opened Jan 11, 2025 by yuanyehome
@pufanyi
ProTip! no:milestone will show everything without a milestone.