Lm-eval with vllm#2217

Closed

ziqing-huang wants to merge 95 commits intomainfrom

ziqing/vllm_cleanup

Commits on Jul 25, 2025

initial
Chi Heem W
committed
Add missing data_utils/count_dataset.py
Chi Heem W
committed

Commits on Jul 26, 2025

linting
Chi Heem W
committed

Commits on Jul 27, 2025

Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed

Commits on Jul 28, 2025

change for loading from checkpoints
Chi Heem W
committed
Linting
Chi Heem W
committed
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed
linting
Chi Heem W
committed
Merge branch 'main' into chiheem/thinking_sft
chiheem
authored
fix import
Chi Heem W
committed
linting
Chi Heem W
committed
linting
Chi Heem W
committed
llama
Chi Heem W
committed
Yet another linting
Chi Heem W
committed
issue with linter
Chi Heem W
committed

Commits on Jul 31, 2025

edit comments
Chi Heem W
committed

Commits on Aug 1, 2025

lint
Chi Heem W
committed

Commits on Aug 15, 2025

Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed

Commits on Aug 18, 2025

Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed

Commits on Aug 24, 2025

Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed

Commits on Aug 28, 2025

Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed

Commits on Sep 2, 2025

helm eval
Chi Heem W
committed

Commits on Sep 4, 2025

add nemotron-v2
Chi Heem W
committed
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed
fix for nemotron-v2
Chi Heem W
committed
refactor for easy import
Chi Heem W
committed

Commits on Sep 5, 2025

streamline
Chi Heem W
committed
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed

Commits on Sep 8, 2025

Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed

Commits on Sep 11, 2025

working evaluator but crashes at levanter not implmented
Chi Heem W
committed
change eval
Chi Heem W
committed

Commits on Sep 12, 2025

add packages, remove levanter lm eval
Chi Heem W
committed

Commits on Sep 17, 2025

fix to lm_evaluation_harness_evaluator
Chi Heem W
committed
change humaneval to humaneval_instruct
Chi Heem W
committed
Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed
add pip for lm_eval_harness
Chi Heem W
committed

Commits on Sep 19, 2025

Resolve merge conflicts in pyproject.toml and uv.lock
Chi Heem W
committed

Commits on Sep 20, 2025

Update uv.lock with latest package versions
Chi Heem W
committed
Fix chex version constraint to use available 0.1.90
Chi Heem W
committed

Commits on Sep 22, 2025

update to conform to new executor framework
Chi Heem W
committed

Commits on Sep 23, 2025

Merge remote-tracking branch 'origin/main' into chiheem/thinking_sft
Chi Heem W
committed
linting
Chi Heem W
committed
linting
Chi Heem W
committed
update required libraries for levanter_lm_eval_evaluator
Chi Heem W
committed

Commits on Sep 24, 2025

test only script
Chi Heem W
committed

Commits on Sep 25, 2025

minor changes
Chi Heem W
committed

Commits on Sep 29, 2025

depreciate trust_remote_code
Chi Heem W
committed
add comments for future reference
Chi Heem W
committed
add comments for future reference
Chi Heem W
committed
Merge remote-tracking branch 'origin/main' into chiheem/levanter_lm_eval
Chi Heem W
committed
cleanup
Chi Heem W
committed

Commits on Sep 30, 2025

test llama-3.1
Chi Heem W
committed
removed eval_batch_size
Chi Heem W
committed

Commits on Oct 1, 2025

changed default settings for levanter_lm_eval_evaluator
Chi Heem W
committed

Commits on Oct 9, 2025

Fix for unsafe code eval + option to suppress logs from ray
Chi Heem W
committed
change eval tag
Chi Heem W
committed
Set lm-eval hash to make uv happy
Chi Heem W
committed
testing config for exp905c
Chi Heem W
committed
try to add max_gen_tok as arg
Chi Heem W
committed

Commits on Oct 12, 2025

Pass remote model path directly to Levanter (requires https://github.com/marin-community/levanter/pull/980)
AlienKevin
committed

Commits on Oct 14, 2025

refactor to use max_length instead of max_gen_tokens
Chi Heem W
committed
clean up control for printing during generation
Chi Heem W
committed
linting
Chi Heem W
committed
allows lm-eval-evaluator to pass apply_chat_template flag
Chi Heem W
committed

Commits on Oct 21, 2025

allow marin to override lm-eval default config
Chi Heem W
committed

Commits on Oct 22, 2025

catch wandb not finishing gracefully
Chi Heem W
committed

Commits on Oct 28, 2025

clean up eval
Chi Heem W
committed
Merge remote-tracking branch 'origin/main' into chiheem/levanter_lm_eval
Chi Heem W
committed

Commits on Oct 31, 2025

wandb fixes
Chi Heem W
committed

Commits on Nov 5, 2025

add qwen2.5-7B
Chi Heem W
committed
disable logging on levanter lm eval
Chi Heem W
committed
Merge remote-tracking branch 'origin/main' into chiheem/levanter_lm_eval
Chi Heem W
committed

Commits on Nov 6, 2025

Merge remote-tracking branch 'origin/main' into chiheem/levanter_lm_eval
Chi Heem W
committed

Commits on Nov 17, 2025

project script changes
Chi Heem W
committed
system changes
Chi Heem W
committed
update pointer to eval harness version
Chi Heem W
committed

Commits on Nov 18, 2025

levanter updates
Chi Heem W
committed

Commits on Nov 24, 2025

update core scripts
Chi Heem W
committed
update pyprojects to use vllm
Chi Heem W
committed

Commits on Dec 1, 2025

almost working vllm lm-eval
Chi Heem W
committed
working vllm lm-eval
Chi Heem W
committed
cleanup
Chi Heem W
committed

Commits on Dec 2, 2025

script to download model from HF to GCS
Chi Heem W
committed

Commits on Dec 3, 2025

try to add generation kwargs
Chi Heem W
committed
uv lock
Chi Heem W
committed

Commits on Dec 4, 2025

more hacking to make it more robust. still fail on aime25
Chi Heem W
committed

Commits on Dec 8, 2025

vllm
ziqing-huang
committed

Commits on Dec 9, 2025

Save changes
ziqing-huang
committed

Commits on Dec 11, 2025

Cleanup
ziqing-huang
committed