[agent] system message + SWE-Bench instruction improvements #7018
+152
−44
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Traceback (most recent call last):
File "/home/xingyaow/OpenHands-dev/evaluation/utils/shared.py", line 323, in _process_instance_wrapper
result = process_instance_func(instance, metadata, use_mp, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xingyaow/OpenHands-dev/evaluation/benchmarks/swe_bench/run_infer.py", line 461, in process_instance
raise EvalException('Fatal error detected: ' + state.last_error)
evaluation.utils.shared.EvalException: Fatal error detected: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/logging/__init__.py", line 1160, in emit
msg = self.format(record)
^^^^^^^^^^^^^^^^^^^
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/logging/__init__.py", line 999, in format
return fmt.format(record)
^^^^^^^^^^^^^^^^^^
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/logging/__init__.py", line 719, in format
s = s + self.formatStack(record.stack_info)
~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TypeError: can only concatenate str (not "bool") to str
Call stack:
File "/home/xingyaow/OpenHands-dev/evaluation/benchmarks/swe_bench/run_infer.py", line 584, in <module>
run_evaluation(
File "/home/xingyaow/OpenHands-dev/evaluation/utils/shared.py", line 415, in run_evaluation
with mp.Pool(num_workers) as pool:
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/pool.py", line 215, in __init__
self._repopulate_pool()
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/pool.py", line 306, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/pool.py", line 329, in _repopulate_pool_static
w.start()
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/context.py", line 282, in _Popen
return Popen(process_obj)
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/popen_fork.py", line 71, in _launch
code = process_obj._bootstrap(parent_sentinel=child_r)
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/xingyaow/micromamba/envs/oh/lib/python3.12/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/xingyaow/OpenHands-dev/evaluation/utils/shared.py", line 384, in _process_instance_wrapper_mp
return _process_instance_wrapper(*args)
File "/home/xingyaow/OpenHands-dev/evaluation/utils/shared.py", line 378, in _process_instance_wrapper
logger.error(msg)
Message: '----------\nError in instance [pydata__xarray-7233]: Fatal error detected: ConnectionError: (\'Connection aborted.\', RemoteDisconnected(\'Remote end closed connection without response\')). Stacktrace:\nTraceback (most recent call last):\n File "/home/xingyaow/OpenHands-dev/evaluation/utils/shared.py", line 323, in _process_instance_wrapper\n result = process_instance_func(instance, metadata, use_mp, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/xingyaow/OpenHands-dev/evaluation/benchmarks/swe_bench/run_infer.py", line 461, in process_instance\n raise EvalException(\'Fatal error detected: \' + state.last_error)\nevaluation.utils.shared.EvalException: Fatal error detected: ConnectionError: (\'Connection aborted.\', RemoteDisconnected(\'Remote end closed connection without response\'))\n\n----------[The above error occurred. Retrying... (attempt 1 of 5)]----------\n'
Arguments: ()
Instance pydata__xarray-7233 - 2025-02-27 23:04:06,218 - WARNING - Using o
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
End-user friendly description of the problem this fixes or functionality that this introduces.
Give a summary of what the PR does, explaining any non-trivial design decisions.
Follow-up of #7002 and #7010.
This PR re-organized the system message and the user manual instructions to make them clearer.
Link of any specific issues this addresses.
To run this PR locally, use the following command: