feat(eval): support DeepSeek Harness and run a reproducible ClawBench benchmark

[reacher-z]

Goal

Add DeepSeek Harness (dsh) as a first-class ClawBench harness, then run and publish a reproducible ClawBench evaluation with it.

DeepSeek Harness is an official DeepSeek open-source agent harness built around an “everything is a plugin” architecture. It is currently in developer preview and may introduce breaking changes, so the integration should pin an exact revision/version.

Scope

1. Harness adapter

• Add a deepseek-harness / dsh adapter alongside the existing OpenClaw, HermesAgent, and Claude Code paths.
• Provide setup and launch scripts, e.g. setup-deepseek-harness.sh and run-deepseek-harness.sh.
• Map ClawBench tasks into the dsh session/task interface.
• Capture the standard ClawBench trace bundle:
  • recording.mp4
  • actions.jsonl
  • agent-messages.jsonl
  • requests.jsonl
  • interception.json
  • run-meta.json
• Record the dsh package version or Git commit, plugin set, model configuration, corpus revision, and ClawBench commit in run-meta.json.
• Preserve ClawBench’s existing timeout, interception, judge, retry, and failure-taxonomy semantics.

2. Reproducibility smoke test

Run a small preregistered subset before the full evaluation:

• 5–10 tasks spanning multiple sites and metaclasses
• One inexpensive supported model/configuration
• Fixed task IDs, corpus commit, dsh revision, judge, limits, and seeds where applicable

Verify that:

• dsh completes tasks end-to-end under ClawBench;
• every attempted task produces a complete or explicitly failed trace bundle;
• failures are attributed to the agent, harness/adapter, site, model/API, or grader infrastructure;
• saved artifacts can be rescored without rerunning the agent.

3. Benchmark run

After the smoke test passes:

• Run the full ClawBench V2 corpus with DeepSeek Harness.
• If budget permits, also run V1 for historical comparability.
• Evaluate at least one model already tested under another harness, allowing the comparison to isolate harness effects.
• Match the model endpoint, task snapshot, judge, per-task limits, retry policy, and evaluation window.
• Run multiple trials, or clearly label the result as single-trial; do not silently mix protocols.

Track at minimum:

• Reward / task success rate
• Stage-1 interception rate
• Stage-2 judged success rate
• Attempted, completed, and infrastructure-failed task counts
• Cost, tokens, and wall-clock time
• Per-task result and failure category

4. Publish results

• Store raw and derived artifacts using the existing eval-results/ conventions.
• Add the DeepSeek Harness result rows to the leaderboard.
• Document the exact reproduction commands and environment.
• Publish a matched-model cross-harness comparison, including confidence intervals or trial variance when available.
• Clearly disclose that dsh is in developer preview and identify the exact evaluated revision.

Suggested CLI

# Smoke test
clawbench run \
  --corpus v2 \
  --harness deepseek-harness \
  --model <matched-model> \
  --tasks <smoke-test-ids>

# Full benchmark
clawbench run \
  --corpus v2 \
  --harness deepseek-harness \
  --model <matched-model>

[liyangbing]

This is a good place to make the DSH adapter evidence-first. Before a full V1/V2 run, I would preregister a small smoke matrix and make the adapter prove its boundaries independently:

• pin the exact DSH source commit or package artifact, Node/pnpm/runtime identity, plugin/profile composition, ClawBench commit, corpus snapshot, model endpoint, judge, timeout and retry policy;
• run 5–10 fixed tasks that exercise ordinary tool calls, an approval/denial path, a timeout or cancellation path, and one task that produces a durable Session artifact;
• require one run-meta record per task, even for setup or infrastructure failure, with secrets and personal paths redacted;
• distinguish agent/model failure, DSH adapter failure, site/interception failure, judge failure, and missing/incomplete trace bundle;
• rescore saved artifacts without rerunning the agent, and verify that a repeated smoke run does not silently mix revisions or retry policies.

For cross-harness comparison, keep reward, interception, judged success, attempted/completed/infrastructure-failed counts, token/cost receipts, wall time, and per-task failure categories separate. A final answer or a single aggregate score cannot prove tool lifecycle, approval, or accounting behavior.

The handbook has an interactive boundary scorecard and a version-evidence checklist that may help define the adapter acceptance gates:

• https://sandbaseai.github.io/deepseek-harness-handbook/agent-harness-scorecard.html
• https://sandbaseai.github.io/deepseek-harness-handbook/version-evidence.html

This is an independent community handbook, not a ClawBench result or an official DeepSeek AI benchmark. Keep alpha/developer-preview runs visibly labeled, and publish the exact evaluated revision before comparing it with another harness.