Skip to content

Lm-eval with vllm #2217

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ziqing-huang wants to merge 95 commits into from
Closed

Lm-eval with vllm #2217

ziqing-huang wants to merge 95 commits into from

Conversation

@ziqing-huang
Copy link
Contributor

@ziqing-huang ziqing-huang commented Dec 11, 2025 •
edited
Loading

Description

Including evals with vllm for #1602 except log probs.

Chi Heem W and others added 30 commits July 25, 2025 23:20
initial
637d445
Add missing data_utils/count_dataset.py
9050e9f
linting
d1b9e8a
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
aa7ab61
change for loading from checkpoints
d261f2d
Linting
2a9362e
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
7bfb00c
linting
7a5575d
@chiheem
Merge branch 'main' into chiheem/thinking_sft
08a17e7
fix import
952c08e
linting
c72270f
linting
95524c5
llama
77995ab
Yet another linting
94a4fad
issue with linter
4cd5d0a
edit comments
49526ed
lint
90043c7
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
3c6a1da
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
1282a57
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
c71cb8b
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
9645c5a
helm eval
3b3120b
add nemotron-v2
e8c10b6
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
fa257bb
fix for nemotron-v2
2e262af
refactor for easy import
4b03ec5
streamline
43cb5f6
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
da493a2
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
77ec56b
working evaluator but crashes at levanter not implmented
eb9c4af
Chi Heem W and others added 26 commits October 14, 2025 22:21
allows lm-eval-evaluator to pass apply_chat_template flag
bc1ff45
allow marin to override lm-eval default config
649dca9
catch wandb not finishing gracefully
e8e1341
clean up eval
9ce2b26
Merge remote-tracking branch 'origin/main' into chiheem/levanter_lm_eval
d884e41
wandb fixes
6f90eb1
add qwen2.5-7B
f80f0e3
disable logging on levanter lm eval
ac75e0d
Merge remote-tracking branch 'origin/main' into chiheem/levanter_lm_eval
7111c7a
Merge remote-tracking branch 'origin/main' into chiheem/levanter_lm_eval
3e4f640
project script changes
895c74f
system changes
083e289
update pointer to eval harness version
9c28152
levanter updates
4f9d33f
update core scripts
4f227b7
update pyprojects to use vllm
36d6158
almost working vllm lm-eval
16c6f35
working vllm lm-eval
3eaad9e
cleanup
c8f572e
script to download model from HF to GCS
59eb810
try to add generation kwargs
6ad5de4
uv lock
a2f6dc4
more hacking to make it more robust. still fail on aime25
20320a4
@ziqing-huang
vllm
32eb045
@ziqing-huang
Save changes
4a3a620
@ziqing-huang
Cleanup
d372eb4
@github-actions
Copy link
Contributor

github-actions bot commented Jan 4, 2026

This pull request has been inactive for 23 days and is marked as stale.
If there is no further activity within 7 days, it will be automatically closed.
If you believe this PR should remain open, please add a comment or update the PR.

@github-actions github-actions bot added the stale label Jan 4, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 12, 2026

This pull request has been automatically closed due to inactivity.
If you would like to continue working on this, please reopen it or create a new PR.

@github-actions github-actions bot closed this Jan 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

Assignees

No one assigned

Labels

stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ziqing-huang @chiheem @AlienKevin