Skip to content

Agent policy legibility, plus the Windows stall bisect #120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
b-macker merged 4 commits into from
Aug 4, 2026
Merged

Agent policy legibility, plus the Windows stall bisect #120

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
056e659
Make agent policy refusals and environment keys legible
claude Aug 3, 2026
d05a65d
Make the agent stub survive a loaded CI runner
claude Aug 3, 2026
79a7f20
Confine the stub retry to POSIX — it hung the Windows job
claude Aug 3, 2026
e5c6f00
Exclude stub-backed suites on Windows — apply a diagnosis already in …
claude Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 42 additions & 9 deletions src/stdlib/agent_impl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -695,6 +695,14 @@ static NaabVal buildEnvironmentDict(int handle_id, const std::string& config_nam
config->max_total_tokens > 0
? std::max(0, config->max_total_tokens - t.input_tokens - t.output_tokens)
: -1);
} else {
// Always emit. A key that vanishes is indistinguishable from one
// holding 0 via dict.get(), so a script cannot tell "no limit
// configured" from "budget exhausted" — the reading that matters
// most. -1 is the sentinel already used above for an unlimited
// token budget, and by escalation_turn below.
state["turns_remaining"] = NaabVal::makeInt(-1);
state["tokens_remaining"] = NaabVal::makeInt(-1);
}
state["challenges_passed"] = NaabVal::makeInt(t.challenges_passed);
state["challenges_failed"] = NaabVal::makeInt(t.challenges_failed);
Expand All @@ -704,6 +712,10 @@ static NaabVal buildEnvironmentDict(int handle_id, const std::string& config_nam
if (t.lease_expires_turn > 0) {
state["lease_remaining"] = NaabVal::makeInt(
std::max(0, t.lease_expires_turn - t.turns));
} else {
// -1, NOT 0: no lease configured is not an expired lease, and 0
// is exactly what an expired one reports.
state["lease_remaining"] = NaabVal::makeInt(-1);
}
// Wall-clock lease: remaining seconds
if (config && config->standing_lease_seconds > 0) {
Expand All @@ -712,12 +724,20 @@ static NaabVal buildEnvironmentDict(int handle_id, const std::string& config_nam
std::chrono::duration_cast<std::chrono::seconds>(elapsed).count());
state["lease_remaining_seconds"] = NaabVal::makeInt(
std::max(0, config->standing_lease_seconds - elapsed_sec));
} else {
state["lease_remaining_seconds"] = NaabVal::makeInt(-1);
}
// Tool execution state
if (t.tool_calls_total > 0 || (config && config->tools_enabled)) {
state["tool_calls_total"] = NaabVal::makeInt(t.tool_calls_total);
state["tool_calls_blocked"] = NaabVal::makeInt(t.tool_calls_blocked);
state["tool_total_latency_ms"] = NaabVal::makeInt(static_cast<int>(t.tool_total_latency_ms));
} else {
// 0 is the honest value here, unlike the lease keys above: tools
// disabled means no calls were made.
state["tool_calls_total"] = NaabVal::makeInt(0);
state["tool_calls_blocked"] = NaabVal::makeInt(0);
state["tool_total_latency_ms"] = NaabVal::makeInt(0);
}
}
}
Expand Down Expand Up @@ -843,6 +863,8 @@ static NaabVal buildEnvironmentDict(int handle_id, const std::string& config_nam
if (config && config->risk_budget > 0) {
state["risk_budget_remaining"] = NaabVal::makeInt(
ge->getRemainingBudget(config_name));
} else {
state["risk_budget_remaining"] = NaabVal::makeInt(-1);
}

// Governance level — enum is NORMAL(0), ELEVATED(1), HIGH(2), CRITICAL(3)
Expand Down Expand Up @@ -1073,10 +1095,14 @@ static NaabVal agentCreate(std::vector<NaabVal>& args) {
}
if (!can_delegate) {
throw std::runtime_error(
"Agent error: agent.create denied — parent agent lacks AGENT_SEND permission\n\n"
fmt::format(
"Agent error: agent.create denied — parent agent '{}' lacks AGENT_SEND\n\n"
" Help:\n"
" - Tool functions cannot create agents unless the parent agent's\n"
" allowed_actions includes AGENT_SEND\n");
" - Tool functions cannot create agents unless the CALLING agent's\n"
" allowed_actions includes AGENT_SEND (here it grants delegation,\n"
" not the ability to be sent to)\n"
" - Add AGENT_SEND to agents.{}.allowed_actions in govern.json\n",
t_tool_agent_context->name, t_tool_agent_context->name));
}
}
}
Expand Down Expand Up @@ -1790,10 +1816,12 @@ static NaabVal agentSend(std::vector<NaabVal>& args) {
}
if (!allowed) {
throw std::runtime_error(
"Agent error: Action matrix does not include AGENT_SEND\n\n"
fmt::format(
"Agent error: Agent '{}' — action matrix does not include AGENT_SEND\n\n"
" Help:\n"
" - This agent's allowed_actions list does not permit sending messages\n"
" - Add AGENT_SEND to the agent's allowed_actions configuration\n");
" - Add AGENT_SEND to agents.{}.allowed_actions in govern.json\n",
config_name, config_name));
}
}
// Telemetry: admission gate passed — exposes exposure tracking state for audit
Expand Down Expand Up @@ -3591,9 +3619,12 @@ static NaabVal agentSend(std::vector<NaabVal>& args) {
gov_engine->emitEvent(governance::RuntimeEventType::CHECK_FAILED,
"agent_restriction:shell_blocked(" + config_name + ")", "", 0);
throw std::runtime_error(fmt::format(
"Agent error: Response from '{}' contains shell commands\n\n"
"Agent error: Response from '{}' contains shell command syntax\n\n"
" Help:\n"
" - Shell execution is blocked for this agent role\n"
" - This agent's shell_allowed is false, which also blocks shell\n"
" syntax appearing in its RESPONSE TEXT — nothing was executed\n"
" - Patterns matched include fenced bash/sh blocks and lines of the\n"
" form '$ <command>', so package-manager instructions trip it too\n"
" - Configure capabilities.shell.enabled or agent shell_allowed in govern.json\n",
config_name));
}
Expand Down Expand Up @@ -4878,9 +4909,11 @@ static NaabVal agentPropose(std::vector<NaabVal>& args) {
if (a == "AGENT_SEND") { allowed = true; break; }
if (!allowed) {
throw std::runtime_error(
"Agent error: Action matrix does not include AGENT_SEND\n\n"
fmt::format(
"Agent error: Agent '{}' — action matrix does not include AGENT_SEND\n\n"
" Help:\n"
" - Add AGENT_SEND to the agent's allowed_actions configuration\n");
" - Add AGENT_SEND to agents.{}.allowed_actions in govern.json\n",
config_name, config_name));
}
}
// Accounting: one logical transition is being attempted (N candidates
Expand Down
50 changes: 46 additions & 4 deletions tests/governance_v4/test_absorption_degenerate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,52 @@ export FAKE_KEY_ABSDEG="fake-key-absorb-degen"
sign_govern() { (cd "$1" && NAAB_SIGNING_KEY="$NAAB_SIGNING_KEY" "$NAAB" --sign-governance >/dev/null 2>&1) || true; }

start_stub() { # $1=fixture $2=workdir
STUB_PORT=$(( (RANDOM % 20000) + 20000 ))
python3 "$SCRIPT_DIR/../helpers/agent_stub.py" "$STUB_PORT" "$1" "$2" > "$2/stub.log" 2>&1 &
STUB_PID=$!
for _ in $(seq 1 50); do grep -q READY "$2/stub.log" 2>/dev/null && return 0; sleep 0.1; done
# Port is picked at random with no bind check, and the readiness wait used to
# be a flat 5s. Both fail on a loaded CI runner: a collision (or a lingering
# TIME_WAIT socket) leaves the stub dead, and python3 startup + bind can
# exceed 5s. Either way every later assertion in the suite fails for a reason
# that has nothing to do with what the test measures. Three consecutive CI
# runs failed this way, each in a DIFFERENT stub-backed suite, none of them
# reproducible locally.
local _fx="$1" _dir="$2" _try _i _tries=3
# POSIX only. Under MSYS2 this retry path is actively harmful, and it is the
# failure path specifically — a stub that comes up promptly never enters it,
# which is why Windows passed until the retry itself was added:
# - `wait` after a plain TERM can block forever. run-all-tests.sh already
# warns that native Windows binaries under MSYS2 ignore TERM and that
# plain `timeout` "can wait forever"; a process tree holding an
# unkillable child is also why the runner could not enforce its own
# step timeout or finalize the step.
# - fork/exec costs ~50-100ms there, and the loop spawns grep + kill +
# sleep per iteration, so 3x300 iterations is dominated by spawning
# rather than by the 30s of intended waiting.
# Windows keeps the single 5s attempt that ran green for many jobs.
case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) _tries=1 ;; esac
[ -n "${WINDIR:-}" ] && _tries=1
for _try in $(seq 1 $_tries); do
STUB_PORT=$(( (RANDOM % 20000) + 20000 ))
: > "$_dir/stub.log"
python3 "$SCRIPT_DIR/../helpers/agent_stub.py" "$STUB_PORT" "$_fx" "$_dir" > "$_dir/stub.log" 2>&1 &
STUB_PID=$!
if [ "$_tries" -eq 1 ]; then
for _i in $(seq 1 50); do
grep -q READY "$_dir/stub.log" 2>/dev/null && return 0
sleep 0.1
done
return 1
fi
# 30s, not 5s — a slow start is not a failed start. Sleep 0.5 keeps the
# spawn count near the original despite the longer ceiling.
for _i in $(seq 1 60); do
grep -q READY "$_dir/stub.log" 2>/dev/null && return 0
kill -0 "$STUB_PID" 2>/dev/null || break
sleep 0.5
done
# SIGKILL, and no unbounded wait: reaping is not worth a hang.
kill -9 "$STUB_PID" 2>/dev/null; STUB_PID=""
done
echo " start_stub: no READY after 3 port attempts — stub log tail:" >&2
tail -3 "$_dir/stub.log" >&2 2>/dev/null
return 1
}
stop_stub() { [ -n "$STUB_PID" ] && kill "$STUB_PID" 2>/dev/null; wait "$STUB_PID" 2>/dev/null; STUB_PID=""; }
Expand Down
77 changes: 73 additions & 4 deletions tests/governance_v4/test_adversarial_detection.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,33 @@ pass() { PASS_COUNT=$((PASS_COUNT + 1)); echo -e " ${GREEN}PASS${NC} [$1] $2";
fail() { FAIL_COUNT=$((FAIL_COUNT + 1)); echo -e " ${RED}FAIL${NC} [$1] $2"; [ -n "${3:-}" ] && echo -e " ${RED}-> $3${NC}"; FAILURES="${FAILURES}\n [$1] $2"; }
skip() { SKIP_COUNT=$((SKIP_COUNT + 1)); echo -e " ${YELLOW}SKIP${NC} [$1] $2"; }

# EXPERIMENT (reversible in one commit). build-windows has stalled inside
# "CLI tests — shell suites" four times: the step sits in_progress ~47 minutes,
# the runner is killed service-side, and the log archive 404s. timeout-minutes
# has now failed to fire TWICE, so the runner cannot enforce its own step
# timeout — we cannot read our way to the cause, only change the outcome.
#
# A live observation caught it hanging at the test_challenge_discrimination.sh
# header, immediately after another stub-backed suite passed. These 9 suites are
# the only ones that run a Python HTTP server and talk to it from a native
# Windows binary under MSYS2, so they are the region to exclude first.
#
# Read the next Windows run as the result:
# green -> these suites are implicated; narrow from 9
# stalls -> they are exonerated; the cause is elsewhere in the phase
#
# Coverage is not lost: build-linux and Build & Test both run every one of these
# in full, and what they test (agent governance semantics) is platform-neutral.
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*)
echo " test_adversarial_detection.sh: SKIPPED (stub-backed; excluded on Windows pending stall bisect)"
exit 0 ;;
esac
if [ -n "${WINDIR:-}" ]; then
echo " test_adversarial_detection.sh: SKIPPED (stub-backed; excluded on Windows pending stall bisect)"
exit 0
fi

source "$SCRIPT_DIR/../helpers/trust_setup.sh"
setup_isolated_trust
STUB_PID=""
Expand All @@ -72,10 +99,52 @@ export FAKE_KEY_ADVERSARIAL="fake-key-adversarial"
sign_govern() { (cd "$1" && NAAB_SIGNING_KEY="$NAAB_SIGNING_KEY" "$NAAB" --sign-governance >/dev/null 2>&1) || true; }

start_stub() {
STUB_PORT=$(( (RANDOM % 20000) + 20000 ))
python3 "$SCRIPT_DIR/../helpers/agent_stub.py" "$STUB_PORT" "$1" "$2" > "$2/stub.log" 2>&1 &
STUB_PID=$!
for _ in $(seq 1 50); do grep -q READY "$2/stub.log" 2>/dev/null && return 0; sleep 0.1; done
# Port is picked at random with no bind check, and the readiness wait used to
# be a flat 5s. Both fail on a loaded CI runner: a collision (or a lingering
# TIME_WAIT socket) leaves the stub dead, and python3 startup + bind can
# exceed 5s. Either way every later assertion in the suite fails for a reason
# that has nothing to do with what the test measures. Three consecutive CI
# runs failed this way, each in a DIFFERENT stub-backed suite, none of them
# reproducible locally.
local _fx="$1" _dir="$2" _try _i _tries=3
# POSIX only. Under MSYS2 this retry path is actively harmful, and it is the
# failure path specifically — a stub that comes up promptly never enters it,
# which is why Windows passed until the retry itself was added:
# - `wait` after a plain TERM can block forever. run-all-tests.sh already
# warns that native Windows binaries under MSYS2 ignore TERM and that
# plain `timeout` "can wait forever"; a process tree holding an
# unkillable child is also why the runner could not enforce its own
# step timeout or finalize the step.
# - fork/exec costs ~50-100ms there, and the loop spawns grep + kill +
# sleep per iteration, so 3x300 iterations is dominated by spawning
# rather than by the 30s of intended waiting.
# Windows keeps the single 5s attempt that ran green for many jobs.
case "$(uname -s)" in MINGW*|MSYS*|CYGWIN*) _tries=1 ;; esac
[ -n "${WINDIR:-}" ] && _tries=1
for _try in $(seq 1 $_tries); do
STUB_PORT=$(( (RANDOM % 20000) + 20000 ))
: > "$_dir/stub.log"
python3 "$SCRIPT_DIR/../helpers/agent_stub.py" "$STUB_PORT" "$_fx" "$_dir" > "$_dir/stub.log" 2>&1 &
STUB_PID=$!
if [ "$_tries" -eq 1 ]; then
for _i in $(seq 1 50); do
grep -q READY "$_dir/stub.log" 2>/dev/null && return 0
sleep 0.1
done
return 1
fi
# 30s, not 5s — a slow start is not a failed start. Sleep 0.5 keeps the
# spawn count near the original despite the longer ceiling.
for _i in $(seq 1 60); do
grep -q READY "$_dir/stub.log" 2>/dev/null && return 0
kill -0 "$STUB_PID" 2>/dev/null || break
sleep 0.5
done
# SIGKILL, and no unbounded wait: reaping is not worth a hang.
kill -9 "$STUB_PID" 2>/dev/null; STUB_PID=""
done
echo " start_stub: no READY after 3 port attempts — stub log tail:" >&2
tail -3 "$_dir/stub.log" >&2 2>/dev/null
return 1
}
stop_stub() { [ -n "$STUB_PID" ] && kill "$STUB_PID" 2>/dev/null; wait "$STUB_PID" 2>/dev/null; STUB_PID=""; }
Expand Down
28 changes: 28 additions & 0 deletions tests/governance_v4/test_cdd_turn0.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,34 @@ set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
NAAB="$SCRIPT_DIR/../../build/naab-lang"

# EXPERIMENT (reversible in one commit). See test_absorption_degenerate.sh, which
# has carried this guard alone since before this change:
# "Stub-backed HTTP tests hang on Windows/MSYS2 due to signal propagation and
# process cleanup issues. Skip entirely — Linux CI validates the behavior."
# That diagnosis was made once and applied to one file out of 29 that launch the
# stub. build-windows has since stalled four times inside "CLI tests — shell
# suites": the step sits in_progress ~47 minutes, the runner is killed
# service-side, and the log archive 404s. timeout-minutes failed to fire TWICE,
# so the runner cannot enforce its own step timeout either — the cause cannot be
# read, only excluded.
#
# Read the next Windows run as the result:
# green -> stub-backed suites are the cause; narrow from here
# stalls -> they are exonerated and the cause is elsewhere in the phase
#
# Coverage is not lost: build-linux and Build & Test run every one of these in
# full, and agent-governance semantics are platform-neutral.
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*)
echo " test_cdd_turn0.sh: SKIPPED (stub-backed; excluded on Windows pending stall bisect)"
exit 0 ;;
esac
if [ -n "${WINDIR:-}" ]; then
echo " test_cdd_turn0.sh: SKIPPED (stub-backed; excluded on Windows pending stall bisect)"
exit 0
fi


if [ -d "/data/data/com.termux/files/usr/tmp" ]; then
_SYSTMP="${TMPDIR:-/data/data/com.termux/files/usr/tmp}"
else
Expand Down
Loading
Loading