Durability net: CI must exercise the built ForgeBox/CommandBox install→serve journey

[bpamiri]

Problem

The ForgeBox/CommandBox install path (box install wheels-base-template → box server start) was published but broken end to end, and the breakage went undetected for weeks because nothing in CI exercises the built packages the way a CommandBox user installs them. ForgeBox publishing is stable-only and asynchronous, so a packaging regression only surfaces in production long after the offending commit lands.

The path-repair campaign fixed the underlying defects:

• Align the ForgeBox wheels-base-template with the LuCLI wheels new template (single source of truth) #3176 — regenerate the base template from cli/lucli/templates/app/ (single source of truth)
• ForgeBox wheels-base-template ships unsubstituted |placeholders| — box server start fails out of the box #3173 — substitute |placeholders| / {{tokens}} to working defaults at build time (box server start was hard-erroring on cfengine:"|cfmlEngine|" — "Invalid slug detected")
• wheels-base-template package omits the db/ directory — tools/build/base/.gitignore /db/** strips it at publish #3174 — keep db/ in the package (the base .gitignore /db/** stripped it at publish)
• wheels-core box.json double-nests installs: box install wheels-core in a wheels new app lands in vendor/wheels/vendor/wheels and never upgrades the running framework #3177 — land wheels-core flat (box.json double-nested it into vendor/wheels/vendor/wheels/)

These are all easy to silently re-break: a placeholder reintroduced into a config override, a db/ exclude added back, a prepare script re-pointed at the demo app.

The durability net

Add a CI leg that builds the base + core packages from the tree (tools/build/scripts/prepare-base.sh + prepare-core.sh), installs them into a CommandBox container the way box install lands them, boots, and asserts the install→serve journey:

• server starts with no manual edits
• GET / is 200
• /wheels/info is 200 in development
• clean URLs work
• db/ ships
• tools/ci/smoke-env.sh passes its reload-gate / no-trace probes in production

Gated on tools/build/**, cli/lucli/templates/app/**, tools/ci/smoke-env.sh, and the workflow file itself.

Acceptance

• A regression in any of the four campaign fixes turns this leg red on the offending PR (verified locally: the unfixed develop artifact exits 1 with "Invalid slug detected" and the connectivity probe fails; the keystone-composed tree passes all six smoke-env.sh probes plus the dev journey).

Refs #3173, #3174, #3176, #3177.

[wheels-bot]

Wheels Bot — Triage

Classified as framework-design (feature/enhancement request).

This issue requests a new CI leg — building the base + core packages from the tree (tools/build/scripts/prepare-base.sh / prepare-core.sh), installing them the way box install lands them, and asserting the install→serve journey — as a durability net against silent regression of the four landed path-repair fixes (#3173, #3174, #3176, #3177). The deliverable is a new GitHub Actions workflow plus its gating config, not a framework defect fix; the underlying defects are already resolved. Because it asks to build new tooling rather than correct observable wrong behavior in a framework layer, it routes here for a design pass before implementation.

<!-- wheels-bot:triage:3201 -->
<!-- wheels-bot:triage-class:framework-design -->

[wheels-bot]

Wheels Bot — Cross-Framework Research

Question

The ForgeBox/CommandBox install path (box install wheels-base-template -> box server start -> GET /) shipped broken end-to-end for weeks because nothing in CI exercises the built/published packages the way a CommandBox user installs them — the published artifacts differ from the source tree (placeholder/token substitution, .gitignore stripping db/, directory restructuring, box.json nesting). Four packaging defects (#3173, #3174, #3176, #3177) were each "easy to silently re-break." The ask: add a CI leg that builds the base + core packages from the tree (prepare-base.sh/prepare-core.sh), installs them into a clean CommandBox container the way box install lands them, boots, and asserts the install->serve journey (server starts unedited, GET / 200, /wheels/info 200 in dev, clean URLs, db/ ships, smoke-env.sh production probes pass). How do other frameworks guard the published-artifact-vs-source gap?

How other frameworks handle this

Framework	Naming	Surface	Trade-offs	Edge cases
Rails	"rails-new-docker" smoke / generator + application tests	rails new -> build the generated Dockerfile -> podman run -> poll GET /up (60x); separate devcontainer leg scaffolds+migrates+runs the generated app's own tests	E2E leg slow/flaky (container, DBs, health-poll retry); kept off the main matrix; in-suite boot uses a once-generated app_template copied per test	Catches "generated app won't boot/serve"; misses the gem-packaging delta — all legs run the generator from the source tree, never gem install a built .gem
Laravel	"skeleton" + "installer" (laravel new)	Skeleton CI tests the source tree in place (composer install -> artisan test); installer CI runs real laravel new (live composer create-project) in a wide PHPxLaravelxOS matrix, daily cron	Skeleton CI fast but false-confidence; installer CI slow/network-bound; cron catches Packagist drift with zero commits	Catches scaffold + .env/key lifecycle hooks; misses boot — asserts exit-0 + vendor/ + .env, never GET /. Smaller delta (no build-time token/strip step)
Django	startproject/startapp, TemplateCommand, check-manifest	Subprocess scaffolding-render tests (.py-tpl->.py, token substitution), incl. remote --template over real HTTP via LiveServerTestCase; MANIFEST guarded by convention (check-manifest, sdist-vs-VCS diff)	Subprocess+LiveServer tests slow/flaky; check-manifest cheap but lives outside the unit suite	Catches render/token bugs + sdist-vs-VCS file omission; misses the wheel-excludes-templates bug (MANIFEST != wheel) and any boot — tests run on the editable repo, never the built artifact
Phoenix	"installer" / integration_test, "code generation tests"	Separate Mix project depends on installer/framework by path:; per-flag matrix (--database mysql/mssql/sqlite3, --umbrella, --no-html) generates into a clean tmp dir, then compile --warnings-as-errors + format --check + runs the generated app's own mix test	Fast via shared prebuilt deps//_build copied in (--no-install) — but that skips real dep resolution; 3 live DB containers add flake	Catches won't-compile / warnings / failing generated suite per adapter; misses real archive install + runtime HTTP boot — "serve" approximated by compile+ExUnit, no GET /
npm / Verdaccio (+1)	"pack test", "smoke test", "e2e install test"; publint, attw, Verdaccio	Two tiers: (1) npm pack --dry-run manifest snapshot + publint/@arethetypeswrong validate entry points resolve inside the tarball; (2) Verdaccio — real npm publish to a throwaway registry -> npm install into a clean scratch dir -> boot + GET /	Tier 1 = seconds, deterministic, every PR; tier 2 = container + real publish/install, gated to release/nightly; the app-boot curl is the flaky part (needs retry-until-ready)	Directly catches the .npmignore/files-strip footgun (exact analog of .gitignore stripping db/), broken paths, bad config; caveat: npm pack is not byte-identical to npm publish (#679) — only a real publish fully reproduces the divergence

+1 justification: the npm ecosystem is picked because it lives the "what you publish != what's in the repo" problem most acutely (build step + files/.npmignore + prepublishOnly), and it has converged on the exact shape this issue proposes — install the built artifact into a clean location, never symlink the source. It also supplies the sharpest design lesson (the pack-vs-publish fidelity ladder) the Rails/Laravel/Django/Phoenix generator-test model glosses over.

Dominant pattern

Every framework has generator/scaffolding tests, but only the npm ecosystem routinely installs the built, published artifact into a clean environment and boots it. Rails, Laravel, Django, and Phoenix all run their generator from the source tree and therefore structurally cannot catch a packaging/build-step regression (Rails rails new from exe/rails, Phoenix path: deps, Django editable install, Laravel skeleton-in-place). They converge on "scaffold -> compile/own-tests"; they diverge on whether they boot at all (Rails' GET /up and Django's LiveServerTestCase do; Laravel and Phoenix stop at exit-code/compile). The npm tier-2 (Verdaccio) pattern is the only one that closes the published-vs-source gap, and the modern consensus there is unambiguous: publish-and-install the artifact, don't symlink. The Wheels proposal is squarely on the strong side of this consensus — and the production smoke-env.sh probe set actually makes it more thorough than any single framework's leg (real boot + GET / + non-dev hardening probes).

Recommended path for Wheels

Adopt the npm tier-2 / Rails-rails-new-docker shape, but reuse the primitives that already exist in-tree rather than build new ones — the gap is wiring, not new tooling. tools/build/scripts/prepare-base.sh + prepare-core.sh already produce the built packages; tools/ci/smoke-env.sh already encodes the six production probes; tools/build/scripts/test-forgebox-package.sh already exists but only inspects box.json/file counts locally (it does not install into a clean container or boot). The new leg composes these against an installed artifact in a clean CommandBox container. This is CI infrastructure (a GitHub Actions workflow + a thin orchestration script), not a CFML API — so the sketch below is a workflow leg, not a vendor/wheels/ function.

# .github/workflows/forgebox-install-journey.yml
name: ForgeBox install to serve journey
on:
  pull_request:
    branches: [develop]
    paths:                                  # gate per the issue's blast radius
      - 'tools/build/**'
      - 'cli/lucli/templates/app/**'
      - 'tools/ci/smoke-env.sh'
      - '.github/workflows/forgebox-install-journey.yml'
jobs:
  install-journey:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # 1. BUILD the packages from the tree (single source of truth) — #3176/#3177
      - run: bash tools/build/scripts/prepare-core.sh  out/wheels-core
      - run: bash tools/build/scripts/prepare-base.sh  out/wheels-base-template

      # 2. INSTALL the BUILT artifacts into a CLEAN dir the way box install lands them.
      #    Install from the local built folders (fast, deterministic) — NOT the source tree.
      - run: |
          mkdir -p clean-install && cd clean-install
          box install "${GITHUB_WORKSPACE}/out/wheels-base-template" --noSave
          # core must land flat under vendor/wheels (regression-guards #3177's double-nest)

      # 3. ASSERT packaging invariants BEFORE boot (cheap, the npm-pack-snapshot tier)
      - run: bash tools/ci/install-journey-assert.sh clean-install
        # checks: db/ present (#3174); no vendor/wheels/vendor double-nest (#3177);
        # no placeholder token (cfmlEngine / {{token}}) leaked into config (#3173)

      # 4. BOOT unedited + assert the dev journey (the Rails GET /up tier)
      - run: cd clean-install && box server start && sleep 5
      - run: |
          base="http://localhost:${PORT:-8080}"
          curl -fsS -o /dev/null "$base/"             # 200, no manual edits
          curl -fsS -o /dev/null "$base/wheels/info"  # 200 in development

      # 5. PRODUCTION hardening probes — reuse the existing six-probe net unchanged
      - run: BASE_URL="$base" bash tools/ci/smoke-env.sh

# Two usage modes worth supporting in the orchestration:
#   local-build install (above)  -> fast, deterministic, catches all four campaign bugs.
#   staging-publish install      -> box install wheels-base-template@<rc> from a
#                                   throwaway/staging ForgeBox; the Verdaccio-tier
#                                   fidelity that also catches registry-side transforms.

Lives in: .github/workflows/forgebox-install-journey.yml plus a thin tools/ci/install-journey.sh orchestrator (mirrors tools/ci/smoke-env.sh's probe style). No vendor/wheels/ change.
Composes with: existing tools/build/scripts/prepare-*.sh (build), tools/ci/smoke-env.sh (the six production probes — called verbatim), and the smoke-env.yml engine-matrix pattern (smoke Lucee 7 + smoke-adobe Adobe 2023) if the journey should run cross-engine.

Cross-engine notes

None at the CFML layer — the deliverable is shell + GitHub Actions YAML, no Lucee/Adobe/BoxLang API surface. One engine consideration carried over from smoke-env.yml: if the install->serve journey is meant to guard the production-mode error-template class (the Adobe-only #3029 family), the leg should boot the installed artifact on both Lucee 7 and Adobe 2023 — the four campaign bugs are engine-agnostic packaging defects, but the production hardening probes (smoke-env.sh) already distinguish engines.

Open questions

• Fidelity ladder (the npm lesson): install from the local built folder (proposed — fast, deterministic, catches all four campaign bugs) vs. publish to a staging/throwaway ForgeBox and box install wheels-base-template@<rc> from the registry (catches registry-side transforms + the publish lifecycle, but adds network flake and a staging namespace). npm's npm pack != npm publish gap (Test 1.4.5 against lucee 5.1 #679) argues a periodic staging-publish leg is worth adding alongside the per-PR local-build leg. Which tier(s) does the team want, and at what cadence (per-PR vs nightly/release)?
• Hard-fail vs soft-fail: should this gate block merges on develop PRs, or start soft-fail (like Oracle in compat-matrix.yml) until it proves non-flaky?
• Boot port/readiness: box server start port allocation + a retry-until-ready loop (every framework's flakiest step) — reuse smoke-env.sh's --connect-timeout/--max-time discipline rather than a bare sleep.
• Engine breadth: Lucee-7-only (fastest) or the Lucee 7 + Adobe 2023 pair from smoke-env.yml?

Confidence: medium

The dominant pattern strongly endorses the proposal — and smoke-env.sh + prepare-*.sh already exist, so this is wiring, not new framework infrastructure. Downgraded from high per the rule "the +1 framework is doing it differently and the trade-off is non-trivial": npm's explicit pack-smoke-vs-Verdaccio fidelity ladder maps onto a real Wheels decision (local-build install vs staging-ForgeBox publish) that a human should confirm before the leg is written.

Sources

• Rails: https://github.com/rails/rails/blob/main/.github/workflows/rails-new-docker.yml | https://github.com/rails/rails/blob/main/.github/workflows/devcontainer-smoke-test.yml | https://github.com/rails/rails/blob/main/railties/test/isolation/abstract_unit.rb
• Laravel: https://github.com/laravel/installer/blob/master/tests/NewCommandTest.php | https://github.com/laravel/laravel/blob/master/.github/workflows/tests.yml | https://github.com/laravel/laravel/blob/master/composer.json
• Django: https://github.com/django/django/blob/main/tests/admin_scripts/tests.py | https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/ | https://pypi.org/project/check-manifest/
• Phoenix: https://github.com/phoenixframework/phoenix/tree/main/integration_test | https://raw.githubusercontent.com/phoenixframework/phoenix/main/integration_test/test/support/code_generator_case.ex | https://raw.githubusercontent.com/phoenixframework/phoenix/main/integration_test/README.md
• npm / Verdaccio: https://www.verdaccio.org/docs/e2e/ | https://docs.npmjs.com/cli/v11/commands/npm-pack/ | https://publint.dev/

[wheels-bot]

Wheels Bot — No fix proposed

This durability net is already implemented and merged, and the issue is closed as completed.

• The requested CI leg ships as .github/workflows/commandbox-install-smoke.yml, landed in ci(build): add CommandBox install->serve smoke leg for the ForgeBox path #3202 (457aa44d8, "ci(build): add CommandBox install->serve smoke leg for the ForgeBox path").
• It matches the research comment's recommended path almost exactly: it builds the base + core packages from the tree (tools/build/scripts/prepare-base.sh + prepare-core.sh), stages them as box install wheels-base-template lands them (base at the app root, wheels-core at vendor/wheels/), boots under the ortussolutions/commandbox image with zero manual edits, and asserts the install→serve journey:
  • dev: GET / 200, /wheels/info 200, clean URL /main/index 200, db/ ships, no placeholder tokens survive into the artifact (guards ForgeBox wheels-base-template ships unsubstituted |placeholders| — box server start fails out of the box #3173/wheels-base-template package omits the db/ directory — tools/build/base/.gitignore /db/** strips it at publish #3174 plus the root-view generation);
  • prod: tools/ci/smoke-env.sh passes its six reload-gate / no-trace probes.
• It is gated on tools/build/**, cli/lucli/templates/app/**, tools/ci/smoke-env.sh, and the workflow file itself — the blast radius the issue specified.

Because the deliverable already exists in the tree and the issue is closed, opening a draft PR would only duplicate merged work. No spec/implementation is warranted.