[examples] fix: support task-shaped SWE-rebench in Claude Code training - #102
[examples] fix: support task-shaped SWE-rebench in Claude Code training#102Mengyuyang wants to merge 2 commits into
Conversation
fecdf50 to
cba2c14
Compare
| "swerebench", | ||
| "nebius/swe-rebench", | ||
| } | ||
|
|
There was a problem hiding this comment.
The canonical evaluator selector is task.name for task-shaped rows and reward.name for legacy rows. Top-level HuggingFace dataset IDs belong to data_source, not to the recipe’s evaluator selector.
Please remove the speculative dataset-ID aliases and route only the supported canonical names (swe_bench and swe_rebench). Unknown names should fail explicitly.
| return [str(value)] | ||
|
|
||
|
|
||
| def _get_task_config(tools_kwargs: dict | None) -> dict: |
There was a problem hiding this comment.
_get_task_config and _get_reward_metadata are small single-use config-access wrappers, and the Mapping → {} coercions can hide malformed sample contracts. Please inline the simple reads and fail clearly at the task/legacy boundary.
| tools_kwargs = extra_info.get("tools_kwargs", {}) | ||
| reward_config = tools_kwargs.get("reward", {}) | ||
| task_config = tools_kwargs.get("task", {}) | ||
| task_config = task_config if isinstance(task_config, Mapping) else {} |
There was a problem hiding this comment.
Again, please avoid coercing a present but malformed task value to {} here. The compatibility bridge should distinguish “field absent” (valid for legacy rows) from “field present with the wrong type” (invalid input that should fail clearly).
|
The task-shaped fallback currently maps task.sandbox to env_config, but canonical SWE-rebench rows do not contain post_setup_cmd. As a result, the legacy runner will skip the future-git-history cleanup that SWEREBenchTask.run() performs before launching the agent. |
165eda3 to
7d688e7
Compare
7d688e7 to
e599e12
Compare
| @@ -268,6 +312,10 @@ async def claude_code_runner( | |||
| ) | |||
|
|
|||
| try: | |||
There was a problem hiding this comment.
[阻塞] Legacy SWE-ReBench parquet 不会执行 history cleanup
当前 runner 在 legacy 分支中将 task_name 设置为 None，因此旧 parquet 中的 tools_kwargs.reward.name == "swe_rebench" 不会触发已有的 _SWE_REBENCH_GIT_CLEAN_HISTORY。
build_reward_context() 已经统一处理了：
task.name → evaluator
reward.name → evaluator
请将 build_reward_context() 的调用提前到 sandbox 创建前，并复用返回的 metadata["evaluator"]、metadata 和 eval_timeout：
metadata, eval_timeout = build_reward_context(tools_kwargs)
evaluator = metadata["evaluator"]
...
if evaluator == "swe_rebench":
await sandbox.exec_shell(
_SWE_REBENCH_GIT_CLEAN_HISTORY,
workdir="/testbed",
)
后续 reward evaluation 直接复用同一组 metadata / eval_timeout。这样 task-shaped 和 legacy parquet 会遵循完全相同的 evaluator 路由。
同时请删除 post_setup_cmd 的读取和执行逻辑；reBench cleanup 是固定的 task/evaluator 行为，不再由数据字段注入。此次不要求新增测试，但请在 PR Validation 中说明 task-shaped 和 legacy 两条路径均已验证。
2 participants
Summary
Fix the Claude Code blackbox training recipe so that it can consume the
task-shaped SWE-rebench dataset already produced and selected by current
Uni-Agent main.
The current Claude Code training script defaults to
swe_rebench_filtered.parquet, while the recipe still reads the legacytools_kwargs.reward/tools_kwargs.envshape and rejects every rewarddata source other than
swe_bench.The canonical SWE-rebench preprocessor stores the task name, metadata, and
sandbox image under
extra_info.tools_kwargs.task. As a result, thedefault training configuration and the recipe data contract are currently
inconsistent.
Changes
tools_kwargs.task, while preserving the legacytools_kwargs.rewardformat.
tools_kwargs.task.uni_agent.tasks.swe_rebench.reward.compute_rewardimplementation.task types.
Owning layer
This is an examples-layer compatibility fix.
Reward policy remains owned by the Task layer. The Claude Code recipe only
selects the existing task-level evaluator, following the same adapter
pattern already used for SWE-bench by #91.
Why this is needed
Current main contains all three of the following:
run_train.shdefaults toswe_rebench_filtered.parquet.uni_agent.tasks.swe_rebench.preprocessemitstools_kwargs.task.tools_kwargs.rewardand rejectsswe_rebench.Therefore, a dataset generated by the canonical preprocessor cannot be
consumed by the default Claude Code training configuration.
Removing the unsupported-source check would not be sufficient: the recipe
must also extract the task metadata and dispatch to the correct existing
reward evaluator.
Compatibility
No public API or dataset migration is required.
Legacy rows using
tools_kwargs.rewardandtools_kwargs.envcontinue towork. Current task-shaped rows are added as a fallback. Existing SWE-bench
evaluation remains unchanged, and unknown task types continue to fail
closed.