PRs being ignored

[darajava]

I’ve submitted 2 PRs but all PRs since the one I submitted have been completely ignored. There have been quite a few evals submitted since (hundreds of hours of development time). Am I just being impatient or is this project not maintained anymore?

[JortsEnjoyer0]

I came here with the same sentiment. There are PRs going all the way back to early March that haven't been processed yet. People are spending days making these evals and their time is being thrown away. It doesn't really inspire further engagement. I have a few good eval concepts, but each eval needs 100 tests minimum and I'm not sure it's worth my time at this point if it's going to take 3 months to maybe have my eval accepted.

[Ein-Tim]

@JortsEnjoyer0 FYI:

• Idea for Evals: Complex, multi-turn instruction-following Evals #632 by @andrew-openai states:

"We're reducing the 100 sample limit on contributions to 15, in order to encourage more one-off samples or handwritten example contributions like this."

[Juan-de-Salgado]

Making the steps of the process a bit more open, and/or providing a means for contributors to track the progress of their submissions, would be an improvement.

[Juan-de-Salgado]

I submitted #613 two weeks ago; unfortunately, Github blocked my account and the PR went invisible for more than a week.

This may be a common occurrence, given Github's practice of punishing first and asking questions later. Please, OpenAI people, be sure that you go over the list repeatedly to catch these cases, instead of just riding the head of the wave.