Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Peer Review Rules #290

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Peer Review Rules #290

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
456686e
Update inference_rules.adoc
mrmhodak Feb 6, 2024
b1cda85
Update inference_rules.adoc
mrmhodak Feb 13, 2024
f795002
Update inference_rules.adoc
mrmhodak Feb 20, 2024
a13c963
Update inference_rules.adoc
mrmhodak Feb 20, 2024
663da6b
Update inference_rules.adoc
mrmhodak Feb 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions inference_rules.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,21 @@ MLCommons shall retain a library of past audit reports and send copies to MLComm

An audit is expected to be completed within a 90 day period. Audits failing to meet this timeline can be requested to be invalidated by the auditee. The final decision to accept such a request will be taken by the Working Group.

=== Assigned Peer Review Process
mrmhodak marked this conversation as resolved.
Show resolved Hide resolved

In addition to general MLCommons review rules outlined https://github.com/mlperf/policies/blob/master/submission_rules.adoc[here], Inference workgroup uses Assigned Peer Review to improve scrutiny of the results. The goal is to ensure that each submission is reviewed. The process, executed by Results Chair, is as follows:

* Each submitter will be assigned another submitter to review.

* Chair will open a github issue against each company that has review assignment. Issues will be closed once reviewers indicate that they had finished their tasks.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the github issue process, but again don't want to prescribe the process into the rules. Perhaps this could go into a 1 pager document off the rules for future reference.

I think it is okay to leave these exact details to the discretion of the chair and working group to adjust as necessary and is practical.


* During the review, reviewers are asked to pay special attention to: (1) results validity, (2) methodology, (3) instructions for reproducibility, and (4) content of json files in systems directory.

* Any issues discovered are to be filed as github issues and resolved as usual. The issues should be filed before the deadline to raise objections.

* Submitters are encouraged to review other submissions beyond their assigned review.


== Scenarios

In order to enable representative testing of a wide variety of inference
Expand Down
Loading