From 28505af9ac397f4bbb034ce48e555af53ffa15c7 Mon Sep 17 00:00:00 2001 From: Natasha Date: Mon, 20 Jul 2026 16:38:27 +0100 Subject: [PATCH] Add Harness Handbook paper --- resources/papers.md | 11 +++++++ .../scoresheets/resource-harness-handbook.md | 30 +++++++++++++++++++ 2 files changed, 41 insertions(+) create mode 100644 rubrics/scoresheets/resource-harness-handbook.md diff --git a/resources/papers.md b/resources/papers.md index eec5a2f..e7952ab 100644 --- a/resources/papers.md +++ b/resources/papers.md @@ -93,3 +93,14 @@ Each entry uses the repository metadata format: resource type, producer, source, - Evidence quality and maturity level: Emerging but influential benchmark environment. - Last checked: 2026-04-29. - Limitations or caveats: Benchmark performance does not guarantee production safety. Treat it as one source of evidence alongside architecture review, runtime controls, and system-specific red teaming. + +### Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, And Editable + +- Resource type: Academic paper. +- Producer or publisher: Wang et al. +- Source link: . +- Relevance to agentic execution security: Describes behaviour-to-code mapping for agent harnesses that construct prompts, manage state, invoke tools, and coordinate execution, which can support security review and change-impact analysis. +- Coverage: Agent harnesses, behaviour localisation, source-code mapping, static analysis, LLM-assisted structuring, progressive disclosure, edit planning, and verification of candidate implementation locations. +- Evidence quality and maturity level: Emerging academic research with experimental evaluation on modification requests from two open-source harnesses. +- Last checked: 2026-07-20. +- Limitations or caveats: The paper focuses on harness readability and edit planning rather than security controls directly. Security value depends on integrating the method into review, testing, policy, and audit workflows. diff --git a/rubrics/scoresheets/resource-harness-handbook.md b/rubrics/scoresheets/resource-harness-handbook.md new file mode 100644 index 0000000..95c6d2e --- /dev/null +++ b/rubrics/scoresheets/resource-harness-handbook.md @@ -0,0 +1,30 @@ +--- +rubric: resource-quality-rubric.md +artefact: Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, And Editable +artefact_url_or_path: https://arxiv.org/abs/2607.13285 +artefact_version: arXiv v1, submitted 2026-07-14 +last_checked: 2026-07-20 +scored_by: Codex +scored_on: 2026-07-20 +rater_count: 1 +--- + +## Scores + +| Criterion | Score (0–3) | Evidence | Notes | +|---|---|---|---| +| Relevance to agentic AI security | 2 | The abstract frames the agent harness as the system layer that constructs prompts, manages state, invokes tools, and coordinates execution, then introduces behaviour-to-code mapping for locating implementation sites. | Adequate: directly relevant to agent runtime review and change-impact analysis, but the paper is software-engineering focused rather than security-control focused. | +| Evidence-based claims | 2 | The paper reports an evaluation on modification requests from two open-source harnesses and distinguishes the proposed Harness Handbook and Behavior-Guided Progressive Disclosure method from the empirical planning results. | Adequate: load-bearing claims are tied to an experimental setup, although full reproduction depends on the paper artefacts and implementation details. | +| Clarity and editorial quality | 2 | The resource clearly defines the problem of behaviour localisation and explains the representation and progressive-disclosure workflow at a level useful to reviewers of agent codebases. | Adequate: the framing is clear, though readers seeking security-specific review procedures must translate it into their own assurance workflow. | +| Recency and ongoing relevance | 3 | The arXiv record shows v1 submitted on 2026-07-14 and records this review's last-checked date as 2026-07-20. | Strong: current research on a live agent-engineering problem. | +| Transparency of limitations | 2 | The abstract scopes the contribution to locating code locations for modification and reports evaluation across two open-source harnesses. | Adequate: the scope boundary is visible, but limitations around security assurance, tool-specific harness designs, and production deployment are not the primary focus. | + +## Aggregate + +- Raw total: **11 / 15** +- Floor rule triggered: no +- Verdict: **Include with reviewer-noted caveats** + +## Reviewer commentary + +The strongest reason to include this paper is that it addresses a practical discoverability gap in agent harnesses: behaviours are often implemented across prompts, state code, tools, and execution middleware. Its weakest dimension for this repository is that it is not primarily a security paper, so readers should treat it as support for inspectability and change-impact review rather than as a standalone control. The score would increase with released artefacts, security-specific examples, or worked guidance showing how behaviour localisation integrates with policy, testing, and audit evidence.