Fix LLMJudgeMetric.evaluate crash inside a running event loop

[Nitjsefnie]

Fixes #71

LLMJudgeMetric.evaluate() drove an async provider's coroutine with asyncio.get_event_loop().run_until_complete(), which raises RuntimeError when called from inside an already-running loop (the async evaluation pipeline) — and the except swallowed it into a silent 0.0 score.

The fix adds _run_coroutine_blocking(): no running loop → asyncio.run() as before; running loop present → the coroutine executes in a worker thread with its own event loop. This mirrors the repo's existing pattern in openagent_eval/cicd/plugin.py (ThreadPoolExecutor + asyncio.run), with get_running_loop() try/except for detection (avoids the get_event_loop() deprecation). I deliberately didn't use asyncio.run_coroutine_threadsafe(coro, loop).result() — it schedules onto the same loop whose thread is blocked in the synchronous evaluate() call, which deadlocks. Happy to rework toward a shared internal async path instead if you'd prefer that shape.

Regression tests cover both contexts; the running-loop test fails on current main with exactly the reported symptom (LLM judge failed: This event loop is already running → score 0.0). Note the message wording differs by Python version (3.13 says "This event loop is already running"; the issue quotes the 3.11/3.12 wording) — the test asserts behavior, not the string.

Gates: uv run pytest tests/unit — 938 passed, 4 skipped (pre-existing chromadb skips).

Generated by Claude Fable 5 (brief, review), Claude Opus 4.8 (implementation)

[himanshu231204]

@Nitjsefnie update your branch , there is merge conflict with the main branch
openagent_eval/metrics/generation/llm_judge.py

[himanshu231204]

/oc review pr

[opencode-agent]

All 4 tests pass. Here's my review:

---

Review: Fix LLMJudgeMetric.evaluate crash inside a running event loop

Files changed: openagent_eval/metrics/generation/llm_judge.py (+37/-5), tests/unit/test_metrics/test_llm_judge.py (+109/-0)

What it does

Adds _run_coroutine_blocking() helper that detects whether an event loop is already running (get_running_loop() try/except) and either uses asyncio.run() (no loop) or farms the coroutine to a worker thread with its own loop (loop present). This replaces the broken get_event_loop().run_until_complete() pattern that silently crashed inside the async pipeline.

What I like

• Correct diagnosis. The old code's RuntimeError was swallowed by the generic except → silent 0.0 score. The fix addresses the root cause.
• Consistent with existing patterns. The ThreadPoolExecutor + asyncio.run approach mirrors cicd/plugin.py:183-186 exactly.
• Thread safety is handled. The PR author correctly notes why run_coroutine_threadsafe would deadlock — the calling thread is blocked in evaluate(), so scheduling onto the same loop would never execute.
• Tests are thorough. 4 tests cover: sync context, running loop (the regression), sync provider, and missing inputs. The running-loop test would fail on the old code.
• No regression for sync providers. The inspect.iscoroutine guard ensures sync providers are unaffected.

Minor nit

The import concurrent.futures inside the function body (line 45) is a style inconsistency — the rest of the file uses top-level imports. But this is a deliberate choice to keep it as a local import (mirroring cicd/plugin.py:181), so it's fine.

Verdict

Approved. Clean fix, well-tested, consistent with existing patterns, and the regression test correctly reproduces the issue. The merge conflict with main should be straightforward to resolve (just the new _run_coroutine_blocking function and the inspect.iscoroutine guard in evaluate()).

opencode session  |  github run

[himanshu231204]

/oc review pr

[opencode-agent]

Review: Fix LLMJudgeMetric.evaluate crash inside a running event loop

Files: openagent_eval/metrics/generation/llm_judge.py (+37/-5), tests/unit/test_metrics/test_llm_judge.py (+109)

Tests: 4/4 passed.

---

What it does

Adds _run_coroutine_blocking() — a helper that detects a running event loop via get_running_loop() try/except — and uses asyncio.run() when no loop exists, or farms the coroutine to a ThreadPoolExecutor worker thread with its own loop when one does. Replaces the broken get_event_loop().run_until_complete() pattern that silently crashed inside the async pipeline.

Positives

• Correct diagnosis. The old RuntimeError was swallowed by the generic except → silent 0.0 score (the exact symptom in metrics(llm_judge): asyncio.get_event_loop().run_until_complete() crashes in async context (same pattern as #49) #71).
• Consistent with existing patterns. The ThreadPoolExecutor + asyncio.run approach mirrors cicd/plugin.py:183-186 exactly.
• No deadlock. run_coroutine_threadsafe would schedule onto the same loop whose thread is blocked — deadlock. Using a worker thread avoids this correctly.
• No regression for sync providers. The inspect.iscoroutine guard is preserved.
• Tests are thorough and targeted. The running-loop regression test fails on the old code (confirmed by the PR author).

No merge conflict

The comment about a merge conflict is stale — 41a7e90 already merged main into the branch. Diff against origin/main is clean.

Verdict

Approved. Clean, minimal fix. Well-tested. Consistent with the existing codebase pattern.

opencode session  |  github run

[github-actions]

🎉 Congratulations @Nitjsefnie!

Your pull request has been successfully merged into main. 🚀

Thank you for contributing to OpenAgentHQ and helping improve the project.

We truly appreciate your contribution and hope to see you back with more amazing PRs!

Happy Open Sourcing! ❤️