Skip to content

test(cli): fail-closed on directoryRejected / empty / unloadable + browserTest exit - #3395

Merged
bpamiri merged 4 commits into
developfrom
cursor/cli-test-exit-fail-closed-eee6
Aug 23, 2026
Merged

test(cli): fail-closed on directoryRejected / empty / unloadable + browserTest exit#3395
bpamiri merged 4 commits into
developfrom
cursor/cli-test-exit-fail-closed-eee6

Conversation

@bpamiri

@bpamiri bpamiri commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Refs #3083 (does not close).

wheels test exited 0 on runner-honesty failures that tools/test-local.sh and tools/ci/run-tests.sh already treat as exit 1 (directoryRejected, bundlesDiscovered == 0). Unloadable *Spec.cfc files got a WARN in displayTestResults but did not set testsFailed. wheels browser test printed Fail/Error then always return "" (LuCLI success).

TDD on this branch:

  1. Helper stubs + boolean specs (red → green)
  2. Throw-seam specs against no-op $throwIf* (red)
  3. Compose $throwIf* from the helpers; runTests / browserTest call them as the only Wheels.TestsFailed sites (green)

Desk accepted $throwIf* seam specs as process-exit PROVEN. Specs call the seams with stub JSON; no live runTests/browserTest HTTP.

BLOCKER 1 — wheels test fail-closed

$cliTestResultFailed owns the boolean. $throwIfCliTestsFailed throws Wheels.TestsFailed when that helper is true. runTests stashes the JSON + unloadable probe, then calls the seam after the report flushes (so the throw is not swallowed as a crashed run).

BLOCKER 2 — wheels browser test fail-closed

$browserTestResultFailed owns the boolean. $throwIfBrowserTestsFailed throws Wheels.TestsFailed when Fail/Error > 0. Both the text path and --format=json call this seam; missing Playwright / unreachable server still return "".

Red → green

Filter / URL: GET http://localhost:8080/wheels/cli/tests?format=json. Isolate cli.lucli.tests.specs.commands.TestExitFailClosedSpec in bundleStats.

Step SHA TestExitFailClosedSpec
Helper red f63ddd719 5 failed / 4 passed
Helper green cecba650e 9 passed
Throw-seam red ae6d7abe0 7 failed / 11 passed — all fail-shape toThrow red (did not throw … Type=[Wheels.TestsFailed]); helper specs + clean-pass notToThrow green
Throw-seam green 8320573b9 18 passed / 0 failed / 0 error

Throw-lock its that went red then green:

  • $throwIfCliTestsFailed — directoryRejected / bundlesDiscovered=0 / specsFailedToLoad>0 / totalFail>0 / totalError>0
  • $throwIfBrowserTestsFailed — totalFail>0 / totalError>0
  • Clean-pass notToThrow on both seams stayed green through both SHAs

HEAD: 8320573b9a0cdb503dc07f5522b6c79c3c73a578

Required CI checks

  • Validate Commit Messages
  • Lucee 7 + SQLite (LuCLI)
  • Verify docs (cli/lucli/**)
Open in Web Open in Cursor 

cursoragent and others added 2 commits August 23, 2026 23:29
Harness for #3083: $cliTestResultFailed currently mirrors runTests'
totalFail+totalError check, and $browserTestResultFailed mirrors
browserTest's always-success return. Specs expect directoryRejected,
bundlesDiscovered=0, unloadable, and browser Fail/Error to fail-closed.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Peter Amiri <peter@alurium.com>
…r Fail

Expand $cliTestResultFailed to match test-local.sh / run-tests.sh
honesty flags, pass the unloadable-spec probe into the exit path,
and throw Wheels.TestsFailed from browserTest when Fail/Error > 0.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Peter Amiri <peter@alurium.com>
@github-actions github-actions Bot added the docs label Aug 23, 2026
cursoragent and others added 2 commits August 23, 2026 23:39
$throwIfCliTestsFailed and $throwIfBrowserTestsFailed are no-ops so
the Wheels.TestsFailed locks for directoryRejected, empty, unloadable,
and browser Fail/Error go red. Helper boolean specs stay green.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Peter Amiri <peter@alurium.com>
Compose $throwIfCliTestsFailed / $throwIfBrowserTestsFailed from the
existing boolean helpers and wire runTests / browserTest to call them
after the report flushes. No parallel bare throw left beside the seams.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Peter Amiri <peter@alurium.com>
@bpamiri
bpamiri marked this pull request as ready for review August 23, 2026 23:55
@bpamiri
bpamiri merged commit ebb48e5 into develop Aug 23, 2026
10 checks passed
@bpamiri
bpamiri deleted the cursor/cli-test-exit-fail-closed-eee6 branch August 23, 2026 23:55
cursor Bot pushed a commit that referenced this pull request Aug 24, 2026
Tip already fail-closes compile-skip via $countSpecsFailedToLoad
and $throwIfCliTestsFailed. Restore those helpers and the WARN copy.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Peter Amiri <peter@alurium.com>
bpamiri added a commit that referenced this pull request Aug 24, 2026
* test(cli): prove-red Hardener SHOULDs 3-6 (unloadable / docs / CommandBox / orphan)

Lock residual gaps on tip ebb48e5: browser unloadable ignore, WARN
soft-fail copy, guides that still teach vacuous exit 0, CommandBox
test runners that swallow failing exits, and unrouted Tests.cfc.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Peter Amiri <peter@alurium.com>

* fix(cli): fail-closed browser unloadable, deprecate CommandBox test, drop orphan Tests.cfc

SHOULD 3 residual: $browserTestResultFailed takes specsFailedToLoad;
WARN no longer says silently skipped. SHOULD 4: guides name
Wheels.TestsFailed. SHOULD 5: CommandBox test runners error() instead
of swallowing TestBox exits. SHOULD 6: delete unrouted Tests.cfc.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Peter Amiri <peter@alurium.com>

* revert(cli): keep #3395 SHOULD 3 seams; do not reopen C1

Tip already fail-closes compile-skip via $countSpecsFailedToLoad
and $throwIfCliTestsFailed. Restore those helpers and the WARN copy.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Peter Amiri <peter@alurium.com>

* test(cli): prove-red C5 — CommandBox error() must not reach testbox run

CommandBox error() does not abort. Specs now require no testbox run
and return immediately after the deprecation error().

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Peter Amiri <peter@alurium.com>

* fix(cli): return after CommandBox test deprecation; drop testbox run

error() does not abort. Each frozen test runner now returns immediately
and no longer invokes testbox run (C5).

Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Peter Amiri <peter@alurium.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants