testing: publish/preview/cache action logic has no automated tests

[mmcky]

Surfaced by the v0.8.0 validation study.

Problem

Only the container images are validated end-to-end (build-containers.yml → test-container.yml / test-containers-lectures.yml). The action logic itself has no automated tests:

• publish-gh-pages
• preview-netlify, preview-cloudflare
• build-jupyter-cache, restore-jupyter-cache
• setup-environment (including the non-container / standard-mode conda-cache path)

Confidence currently rests on code review plus observed production publishes in the lecture repos. TESTING.md documents container validation and manual local fixtures — not action-level tests.

Why it matters

The publish and cache paths are what consumers depend on most, and a regression there wouldn't surface until a real publish or cache build fails. This is the same tooling layer whose gaps hid #28 (see #83).

Suggested fix

• Add action-level smoke tests — a minimal fixture repo/workflow exercising at least publish-gh-pages and the cache actions.
• Fold in the env-test harness proposed in Docs: guidance on composite actions vs reusable workflows (+ env-test harness proposal) #29.

Context: QuantEcon/lectures#11.

[mmcky]

Design proposal: a two-part harness, not one

This issue's suggested fix is "a minimal fixture repo/workflow". This is a concrete proposal for its shape, prompted by the v0.9.0 release — where the only consumer-facing change (#104) had to be verified with a throwaway harness on a branch, because nothing in CI exercises the cache actions at all.

The short version: one harness can't do this job, because of a structural constraint in how our own actions call each other. Two cheap harnesses can, and they catch different things.

The constraint that shapes everything

GitHub does not allow expressions in uses:. A ref cannot be parameterised, ever.

That matters because build-jupyter-cache calls its siblings at a pinned ref — setup-environment@v0 at action.yml:132, and build-lectures@v0 at :147, :156 and :165. That pinning is correct and deliberate — it's the H8 fix from v0.8.0, and it exists because relative ./ paths in a composite action resolve against the consumer's workspace, not this repo.

The consequence: a test repo pinned at @main still executes @v0 for everything build-jupyter-cache calls internally. That chain cannot be tested before release, by construction. No harness design gets around it, so the design should stop trying and split the work instead.

The proposal

A. A PR-time harness, in this repo, using ./ local paths. Fast, synthetic, runs on every PR that touches an action. This is a generalisation of the throwaway harness used to verify #104: seed a fixture, invoke the action, assert on its outputs and on the filesystem, fail the job on any mismatch. It caught the #104 defect and a control run against the pre-fix code proved it detects it rather than merely passing.

B. A post-release canary, in a separate QuantEcon/test-lectures repo pinned at @v0. Runs on a schedule and immediately after each tag move. This is the only thing that exercises the real consumer path, including the sibling-pinned chain that A structurally cannot reach.

Action	A: PR harness	B: canary
setup-environment	✅	✅
build-lectures	✅	✅
restore-jupyter-cache	✅ both modes, both cache-types	✅ real branch/PR cache scoping
build-jupyter-cache	⚠️ runs, but its internal calls execute @v0, not the PR	✅
preview-netlify, preview-cloudflare	❌ needs a real deploy target	✅
publish-gh-pages	❌ would publish this repo's own Pages	✅

Why B has to be a separate repo

The decisive reason is the one above: in this repo the workspace is the actions repo, so the exact condition that breaks for consumers is the condition that works in an in-repo test. The bug class is invisible here.

Two practical reasons follow. publish-gh-pages needs a real gh-pages branch and the preview actions need real deploy targets — neither of which should point at the actions repo. And the cache actions' semantics that matter most (branch-scoped caches, main→PR inheritance) only exist where there is a real main branch and real PRs.

The fixture is where the current gap is worst

The existing minimal fixture at containers/quantecon/tests/minimal-jupyter-book/_config.yml cannot fail in the ways that matter — already noted in #108:

Setting	Effect
execute_notebooks: "off"	No code cell ever runs, so the kaleido/Chrome stack is never exercised — the exact failure mode behind the v0.9.0 kaleido<1.0 pin
html_theme: sphinx_book_theme	The pinned quantecon-book-theme is never imported
fontpkg: ""	Suppresses the FreeSerif path that the .otf-symlink workaround in the Dockerfile exists for

A fixture worth having deliberately exercises the three things that have actually broken in production: a plotly cell that forces static-image export, a matplotlib cell, and the real quantecon_book_theme under xelatex. Keep it to roughly five lectures so a build is 2–3 minutes rather than the 107 minutes the real-repo matrix takes.

If the canary consumes templates/cache.yml, templates/ci.yml and templates/publish.yml verbatim, it also validates the shipped templates — which #109 flags as having drifted from the shipped code. Two birds.

Suggested staging

Each stage is independently useful, so this doesn't have to land as one lift.

1. PR harness for the cache actions — no new repo, no secrets, generalises what already exists as a throwaway. Closes the largest gap for the least work.
2. Create test-lectures with the fixture and the templates verbatim, wiring cache.yml on main and ci.yml on PR. Covers the build-jupyter-cache chain and real cache scoping.
3. Add preview and publish coverage — this is the stage that needs secrets (Netlify token and site, Cloudflare token, a gh-pages branch).
4. Trigger the canary on tag push, so every release is smoke-tested against a real consumer before the lecture repos see it.

What this still won't cover

Worth stating plainly so the harness isn't over-trusted. It won't cover pre-release testing of the build-jupyter-cache sibling chain — that's the structural constraint, and stage 4 mitigates it by shortening the window rather than closing it. It won't cover real-scale performance or content diversity, so test-containers-lectures.yml against the three real lecture repos stays as-is. And the lecture-jax GPU path remains out of scope while QuantEcon/lecture-jax#284 is open.

Why this is worth the maintenance

The evidence is specific rather than general. restore-jupyter-cache's save-cache: true mode shipped dead on arrival in v0.8.0 — every output empty, fail-on-miss failing unconditionally — and survived until a maintenance audit found it eight months later (#104). A canary running the save-cache path would have caught it the same day. #105 (preview deploy errors swallowed) and #34/H4 are the same shape: bugs in the layer with no tests, found by reading rather than by failing.

A secondary benefit worth noting: a canary makes the "repin consumers" step safe. Repin the canary first, watch one real build, then repin the lecture repos — instead of lecture-dp being the thing that discovers a regression.