Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
96 Open 1,247 Closed
96 Open 1,247 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Add --examples Argument for Fine-Grained Task Evaluation in lm-evaluation-harness. This feature is the first step towards efficient multi-prompt evaluation with PromptEval [1,2]
#2520 opened Nov 26, 2024 by felipemaiapolo Loading…
1
make utility function to handle until
#2518 opened Nov 26, 2024 by baberabb Loading…
Filters bugfix
#2517 opened Nov 26, 2024 by baberabb Loading…
1
max_length not used
#2515 opened Nov 25, 2024 by lintangsutawika Loading…
3
Update Unitxt task to use locally installed unitxt and not download Unitxt code from Huggingface
#2514 opened Nov 25, 2024 by yoavkatz Loading…
Added small fix to split by eos_token_id before decoding
#2512 opened Nov 24, 2024 by EtashGuha Loading…
1
AraDICE task config file
#2507 opened Nov 19, 2024 by firojalam Loading…
1
fixed mmlu generative response extraction
#2503 opened Nov 18, 2024 by RawthiL Loading…
Added regex filter for bbh fewshot
#2502 opened Nov 18, 2024 by RawthiL Loading…
1
Add GigaChat API
#2495 opened Nov 15, 2024 by seldereyy • Draft
Yaml crowspairs tasks
#2488 opened Nov 14, 2024 by NAM00 Loading…
1
Biology ds
#2486 opened Nov 13, 2024 by deema-A Loading…
2
MILU dataset from AI4Bharat for Indic LLM eval
#2482 opened Nov 12, 2024 by abhinand5 Loading…
1
Update citation
#2474 opened Nov 8, 2024 by Sypherd Loading…
1
Use global filter alias
#2473 opened Nov 8, 2024 by Sypherd Loading…
allow fewshots for multimodal tasks
#2450 opened Nov 1, 2024 by artemorloff Loading…
2
Add Aggregation for Kobest Benchmark
#2446 opened Oct 31, 2024 by tryumanshow Loading…
fix tmlu tmlu_taiwan_specific_tasks tag
#2420 opened Oct 22, 2024 by nike00811 Loading…
3
Add YandexGPT API
#2419 opened Oct 21, 2024 by almasgarriev Loading…
1
Fix Type Hints for vLLM CausalLM model
#2408 opened Oct 18, 2024 by qthequartermasterman Loading…
1
Update citation links to Zenodo and DOI to 0.4.5
#2391 opened Oct 9, 2024 by LSinev Loading…
log_samples File name too long. Need truncation or override #1454 FIX
#2384 opened Oct 5, 2024 by kroc99 Loading…
1
add Russian mmlu
#2378 opened Oct 3, 2024 by tatiana-iazykova Loading…
4
Add the BlueBench benchmark
#2369 opened Oct 1, 2024 by shachardon Loading…
3
MMLU Pro Plus
#2366 opened Sep 30, 2024 by asgsaeid Loading…
1
ProTip! Follow long discussions with comments:>50.