Skip to content

fix: change default judge_backend from openclaw to api - #357

Merged
olearycrew merged 1 commit into
mainfrom
fix/judge-api-backend
Apr 24, 2026
Merged

fix: change default judge_backend from openclaw to api#357
olearycrew merged 1 commit into
mainfrom
fix/judge-api-backend

Conversation

@ScuttleBot

Copy link
Copy Markdown
Contributor

Problem

The openclaw backend spins up a full OpenClaw agent with tools for judging. The agent was going rogue — running exec commands and exploring the filesystem instead of just returning JSON scores.

Evidence from live benchmark (Kimi K2.6):

  • 128 'Failed to parse judge JSON response' warnings
  • Judge transcripts show calls to find, ls, exploring workspace
  • Model scored 0.8% (1/123) because judge couldn't parse anything
  • Only automated tasks (no judge) passed

Solution

Change the default judge_backend from openclaw to api.

The api backend does direct API calls without spawning an agent with tools. This is more reliable for the simple scoring task.

Files Changed

  • scripts/lib_grading.py — Change default parameter in grade_task() and _grade_llm_judge()

Testing

Need to run a benchmark with this fix to verify judge responses are now parseable JSON.

The openclaw backend spins up a full OpenClaw agent with tools for judging.
The agent was going rogue - running exec commands and exploring the filesystem
instead of just returning JSON scores. This caused 'Failed to parse judge JSON
response' errors and 0% scores on most tasks.

The api backend does direct API calls without agent tools, which is more
reliable for the simple scoring task.

Root cause: Judge transcripts showed the agent calling find, ls, etc. instead
of outputting scores. 128 parse failures in a single run.
@kilo-code-bot

kilo-code-bot Bot commented Apr 22, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Solid, targeted fix. The change correctly addresses the root cause — switching the default judge_backend from openclaw to api in both grade_task() and _grade_llm_judge() eliminates the rogue agent behavior causing JSON parse failures. The two-line change is minimal and consistent.

Files Reviewed (1 file)
  • scripts/lib_grading.py

Fix any issues in Kilo Cloud


Reviewed by claude-4.6-sonnet-20260217 · 73,409 tokens

@olearycrew
olearycrew merged commit c666177 into main Apr 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants