Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Runtime Bugets and Step Hints #838

Open
wants to merge 20 commits into
base: dev
Choose a base branch
from
Open

Update Runtime Bugets and Step Hints #838

wants to merge 20 commits into from

Conversation

fsschneider
Copy link
Contributor

Depends on #830, #831, #832, and #833.

This PR updates the runtime budgets and step hints according to our working group discussion and issue #836.

For runtimes and step hints, I have always rounded up to the nearest integer.

The new budgets are:

External tuning

  • Criteo 1TB: 7,703s (was 7,703s, no reduction)
  • fastMRI: 4,430s (was 8,859s, reduced to 50%)
  • ResNet: 66,159s (was 63,008s, extended by 5%)
  • ViT: 69,768s (was 77,520s, reduced to 90%)
  • Conformer: 58,015s (was 61,068s, reduced to 95%)
  • DeepSpeech: 44,405s (was 55,506s, reduced to 80%)
  • OGBG: 12,011s (was 18,477s, reduced to 65%)
  • WMT: 43,336s (was 48,151s, reduced to 90%)

Self-tuning
Change the factor from 3x to 1.5x. This keeps the baseline and Schedule-free AdamW still within finite workload scores (except ResNet, which could not be achieved, even with the 3x budget).

@fsschneider
Create equivalent pyproject toml
8327283
@fsschneider
yapf requires toml
c8dc704
@fsschneider
Revert to auto-finding packages (includes tests/)
50658bc
@fsschneider
Match version to GH
616a0f4
@fsschneider
Let setuptools_scm handle versioning.
bad76f5
@fsschneider
Fix version test
ff4a457
@fsschneider
Match file name of version test to the other tests
f97c880
@fsschneider
Fix linting
f98b554
@fsschneider
Update version test to only check major and minor elements, excluding…
8171a32 
… patch version.
@fsschneider
Rename job in regression tests workflow from criteo_resnet_pytorch …
96cc471 
…to `criteo_embed_init_pytorch` due to likely typo.
@fsschneider
Fix some markdown linting issues.
230bf84
@fsschneider
Add trailing new line
ce44582
@fsschneider
Rename package from algorithmic-efficiency to algoperf.
37f556d
@fsschneider
Fix linting (due to shorter package name in imports)
bc666a7
@fsschneider
Update documentation with new targets
59126ae
@fsschneider
Use 1.5 instead of 3x the budget for self-tuning
ff0086c
@fsschneider
Update max allowed runtimes for each workload
03bc79e
@fsschneider
Clarify in comment that its the old budgets
16eb8d6
@fsschneider
Adapt step hint as well
d3f788d
@fsschneider
Clarify step_hint
b3dec67
@fsschneider fsschneider requested a review from a team as a code owner February 3, 2025 11:22
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 3, 2025

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@fsschneider fsschneider added the 🛑 AlgoPerf Leaderboard Blocking rolling AlgoPerf Leaderboard label Feb 3, 2025
@fsschneider fsschneider mentioned this pull request Feb 3, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@priyakasimbeg priyakasimbeg Awaiting requested review from priyakasimbeg

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
🛑 AlgoPerf Leaderboard Blocking rolling AlgoPerf Leaderboard
Projects
None yet
Milestone
Support for Rolling Leaderboard
Development

Successfully merging this pull request may close these issues.
1 participant
@fsschneider