Possible underestimated pass@3 results

[aorwall]

I have evaluated your predictions using my Docker based swe-bench evaluator. I achieve 26% on pass@3 compared to the 22% you reported. It might be worthwhile to review the logs for the failed benchmarks to see if your agent can actually achieve even better results :D

You find the logs and report here

And here's a sheet I use to compare the results.

[zhiyufan]

Thank you for this detailed report!
I quickly went through the comparison sheet you shared and noticed that there are many opposite acr-2/3 results in the two evaluation environments. Maybe you mistakenly mixed up the acr-2/3 results?

[aorwall]

You're right. Good catch. That explains why it looked like some of the benchmark instances failed in my benchmark. Now it covers all the cases you resolved.

[yuntongzhang]

This looks great! The dockerized evaluation environment would be very useful for obtaining consistent evaluation results. We will also try out your docker-based evaluation on our machines :)