Skip to content

Commit 2cb23de

Browse files
committed
fix(eval): parse code fences like lib.sh in the outputs eval
test-convert-outputs.sh's Python fence model read fences only at column 0. With an indented opening fence and a column-0 closer - valid CommonMark that GitHub renders, lint accepts and convert_openclaw splits correctly - the model missed the opener and treated the closer as its own opener, so it reported a torn fenced block across SOUL.md/AGENTS.md and failed the Check Tools workflow on an agent file that was fine. Mirror lib.sh's fence_open_p / fence_closes_p (up to three spaces, same character, a closer at least as long as the opener with nothing but whitespace after the run) so the eval sees the same blocks the converter does. test-convert-fence-model.sh builds a throwaway repo with one such fixture and runs the real eval; on the previous model it fails with the false torn-block report, and with this change all tools validate. The full roster still passes with zero manifest drift.
1 parent 8329468 commit 2cb23de

3 files changed

Lines changed: 81 additions & 4 deletions

File tree

‎.github/workflows/check-tools.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@ jobs:
3636
# maintainers regenerate at landing) and enforced on pushes to main.
3737
run: bash scripts/test-convert-outputs.sh ${{ github.event_name == 'pull_request' && '--drift=advisory' || '' }}
3838

39+
- name: Parse indented code fences like lib.sh in the outputs eval
40+
run: bash scripts/test-convert-fence-model.sh
41+
3942
- name: Validate agent selection (install.sh --agent / --agents-file)
4043
run: bash scripts/test-agent-selection.sh
4144

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
#!/usr/bin/env bash
2+
#
3+
# test-convert-fence-model.sh — the converted-outputs eval must parse source
4+
# code fences the way lib.sh and GitHub do.
5+
#
6+
# The eval's Python model read fences only at column 0, while CommonMark (and
7+
# lib.sh's fence_open_p / fence_closes_p, aligned to it in #855 and #1028)
8+
# allows up to three spaces of indentation and does not constrain a closer by
9+
# the opener's indent. With an indented opening fence and a column-0 closer,
10+
# the model missed the opener and treated the closer as its own opener, so it
11+
# reported the following lines as a torn fenced block across SOUL.md/AGENTS.md
12+
# — a hard failure in the Check Tools workflow for an agent file that lint,
13+
# GitHub and convert_openclaw all handle correctly.
14+
#
15+
# Builds a throwaway repo around the real eval and one such fixture, then
16+
# requires the eval to validate it cleanly. `--update` keeps the check on a
17+
# fresh repo without a committed manifest.
18+
#
19+
# Usage: ./scripts/test-convert-fence-model.sh
20+
21+
set -euo pipefail
22+
23+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
24+
scratch="$(mktemp -d "${TMPDIR:-/tmp}/agency-fence-model.XXXXXX")"
25+
trap 'rm -rf "$scratch"' EXIT
26+
27+
mkdir -p "$scratch/repo/scripts" "$scratch/repo/engineering"
28+
cp "$SCRIPT_DIR/convert.sh" "$SCRIPT_DIR/lib.sh" "$SCRIPT_DIR/build-hermes-plugin.py" \
29+
"$SCRIPT_DIR/test-convert-outputs.sh" "$scratch/repo/scripts/"
30+
31+
cat > "$scratch/repo/divisions.json" <<'EOF'
32+
{
33+
"divisions": {
34+
"engineering": {"label": "Engineering", "icon": "Code", "color": "#3B82F6"}
35+
}
36+
}
37+
EOF
38+
39+
cat > "$scratch/repo/engineering/fence-fixture.md" <<'EOF'
40+
---
41+
name: Fence Model Fixture
42+
description: Fixture agent with an indented opening code fence
43+
color: blue
44+
---
45+
## Identity
46+
```text
47+
code sample
48+
```
49+
## Core Mission
50+
mission text
51+
EOF
52+
53+
cd "$scratch/repo"
54+
if ! bash scripts/test-convert-outputs.sh --update > "$scratch/eval.log" 2>&1; then
55+
cat "$scratch/eval.log" >&2
56+
echo "FAIL: the outputs eval rejected a GitHub-valid indented opening fence" >&2
57+
exit 1
58+
fi
59+
if grep -q 'torn across' "$scratch/eval.log"; then
60+
echo "FAIL: the outputs eval reported a torn fence for correctly split output" >&2
61+
exit 1
62+
fi
63+
echo "PASS: the outputs eval parses indented code fences like lib.sh and GitHub"

‎scripts/test-convert-outputs.sh‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -330,7 +330,14 @@ elif not colour_bad:
330330
# block in half and leaves each file holding a dangling fence, which renders as
331331
# broken markdown for every user of that integration (#849). So every fenced
332332
# block in a source must land intact in exactly one of the two files.
333-
SPLIT_FENCE = re.compile(r"^(`{3,}|~{3,})(.*)$")
333+
#
334+
# This model must read fences exactly as lib.sh's fence_open_p / fence_closes_p
335+
# do (and as GitHub renders): up to three spaces of indentation, same character,
336+
# a closer at least as long as the opener with nothing but whitespace after it.
337+
# Reading fences at column 0 only missed an indented opener and paired its
338+
# column-0 closer with the following lines, reporting a tear the converter had
339+
# not made.
340+
OPEN_FENCE = re.compile(r"^( {0,3})(`{3,}|~{3,})")
334341
335342
def body_lines(text):
336343
"""Mirror lib.sh's get_body, including `$(...)`'s trailing-newline strip."""
@@ -346,13 +353,17 @@ def body_lines(text):
346353
return out
347354
348355
def fence_blocks(lines):
349-
"""Inclusive (opener, closer) index pairs; closer = last line if unterminated."""
356+
"""Inclusive (opener, closer) index pairs; closer = last line if unterminated.
357+
358+
Mirrors lib.sh fence_open_p / fence_closes_p so the eval sees the same
359+
blocks the converter does.
360+
"""
350361
res, marker, mlen, start = [], "", 0, None
351362
for i, line in enumerate(lines):
352-
m = SPLIT_FENCE.match(line)
363+
m = OPEN_FENCE.match(line)
353364
if not m:
354365
continue
355-
tok, rest = m.group(1), m.group(2)
366+
tok, rest = m.group(2), line[m.end():]
356367
if not marker:
357368
marker, mlen, start = tok[0], len(tok), i
358369
elif tok[0] == marker and len(tok) >= mlen and not rest.strip():

0 commit comments

Comments
 (0)