Skip to content

Issues: mlcommons/modelbench

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

206 Open 179 Closed
206 Open 179 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

French runs: pull results for vendors chore Thing that needs doing
#837 opened Feb 1, 2025 by rogthefrog
@rogthefrog
2
Standardize scoring (% safe vs % unsafe)
#833 opened Jan 30, 2025 by rogthefrog
Improve external resource provisioning
#832 opened Jan 30, 2025 by rogthefrog
GOAL: 1.0 Hindi benchmarks milestone A goal that other cards add up to
#831 opened Jan 30, 2025 by wpietri
GOAL: 1.0 Chinese benchmarks milestone A goal that other cards add up to
#830 opened Jan 30, 2025 by wpietri
Run 1.0 French benchmarks
#829 opened Jan 30, 2025 by wpietri
Add hazard info to journal
#823 opened Jan 29, 2025 by bkorycki
calibrate French for new reference model selection
#822 opened Jan 29, 2025 by wpietri
GOAL: 1.0 French Benchmarks milestone A goal that other cards add up to
#820 opened Jan 27, 2025 by wpietri
Goal: French 1.0 benchmarks milestone A goal that other cards add up to
#819 opened Jan 27, 2025 by wpietri
Add evaluator type to hazard UIDs
#816 opened Jan 24, 2025 by bkorycki
modelbench should not accept degenerate or surprising option combinations
#814 opened Jan 23, 2025 by rogthefrog
@rogthefrog
1
Try to replicate production failures using Docker shenanigans
#811 opened Jan 22, 2025 by wpietri
French calibration - full prompt set
#810 opened Jan 22, 2025 by rogthefrog
@rogthefrog
2
Preserve existing run_data in standards file when regenerating the file
#805 opened Jan 18, 2025 by rogthefrog
@rogthefrog
1
update or kill PyPi releases
#804 opened Jan 17, 2025 by wpietri
Version number is off
#803 opened Jan 17, 2025 by wpietri
Start measuring how vendor-hosted model scores change over time
#802 opened Jan 16, 2025 by wpietri
Update supported Python versions
#800 opened Jan 15, 2025 by rogthefrog
@rogthefrog
1
Make modelbench anonymous (offline) practice runs use the 10% extract instead of the full practice set.
#798 opened Jan 15, 2025 by bollacker
1
UI to access benchmarks
#796 opened Jan 14, 2025 by wpietri
run the french models (first pass)
#795 opened Jan 14, 2025 by wpietri
1
For common models, document their frenchness
#794 opened Jan 14, 2025 by wpietri
French calibration - first pass
#793 opened Jan 14, 2025 by wpietri
@rogthefrog
2
Test switching from threads to processes
#792 opened Jan 14, 2025 by wpietri
ProTip! Exclude everything labeled bug with -label:bug.