Suggestion: PR edit limits/review order

[adawginton]

It seems like submissions being reviewed in PR # order could encourage edit camping where you can continuously update a PR with new results and keep your old position in the PR # review queue.

If @0hq @valerio-oai want to keep reviewing submissions in PR # order then I think it would be reasonable to impose a maximum bpb change within which you can edit an original PR's results. If changes are greater than a certain bpb threshold i.e. 0.002 you would have to open up a new PR and get a new spot in the review queue.

[valerio-oai]

This is a reasonable concern, and one that we've thought about. I am quite suspicious of PRs with >1-5 commits, especially over a long period amount of time, and would ask someone to make a new PR if I thought they were messing with this. Thanks for the note!

[asuramaya]

I've been doing this, I did not realize it was frowned upon. I figured it made more sense than flooding the pr queue with more prs.

[anthony-maio]

I closed my earlier PRs and made new ones - no one was looking at them anyway. I don't understand the order this thing is going in at all, but it's not PR #.

[fielding]

I updated some of mine, because I realized non-record submissions still needed al 3 seed runs, so I added them, but it didn't change my results. Also, they are non-record submissions, so hopefully it doesn't matter. I will avoid updating any submissions, though. if possible.

With that said, I think the ones that open a PR and then edit it to something wildly different than the initial title/content will be pretty obvious as long as reviewers keep it in mind and look at the initial title/edit notifications in the PR. Github has us covered on the edit audit log haha