Run unit tests in parallel lanes with a manifest and a ledger - #148
Open
maralcbr wants to merge 1 commit into
Open
Run unit tests in parallel lanes with a manifest and a ledger#148maralcbr wants to merge 1 commit into
maralcbr wants to merge 1 commit into
Conversation
test/all used to run every test serially under set -e, so one slow pty test held everything and a hung test hung the run forever. Replace the loop with test/run-manifest.py: - Two lanes. test/parallel-safe.tests lists tests that only touch their own throwaway directory and run in a thread pool; test/serial.tests lists tests that own something global (a pty, a port, the docker socket) and run one at a time afterwards. Each serial entry says what the global thing is. - Every file under test/unit/ must appear in exactly one manifest or the runner refuses to start (exit 2). A forgotten test is an error, not a silent skip. - Per-test wall-clock timeout (OMARCHY_TEST_TIMEOUT_SECONDS, 300 by default). Each test runs in its own session so a timeout or failure kills the whole process group, and a test that passes but leaves a background process behind is failed. - Python modules run one per process with test/unit on PYTHONPATH and fail closed when zero cases collect. - A JSON ledger (test-runs/unit-test-results.json) is written as "incomplete" before the run and finalised after, and a flock lease stops two runs from interleaving. Exit status: 0 all passed, 1 any failure/timeout/cancellation (fail-fast, as before), 2 the runner refused to start. Add a test workflow that runs test/all in an archlinux container on push and pull_request, since the tests reach for parted, util-linux script, bsdtar and localectl at the versions that ship on the ISO.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace the serial loop in
test/allwith a lane-based runner. Tests that only touch their own temp directory run in parallel; tests that own something global (a pty, a port, the docker socket) run one at a time afterwards. Every test undertest/unit/must be listed intest/parallel-safe.testsortest/serial.tests, or the runner refuses to start, so a forgotten test is an error rather than a silent skip.What changed
test/run-manifest.py: the runner. Thread-pool parallel lane, then the serial lane. Per-test wall-clock timeout (OMARCHY_TEST_TIMEOUT_SECONDS, default 300). Each test runs in its own session, so a timeout kills the whole process group, and a test that passes but leaves a background process behind is failed. Python modules run one per process and fail closed when zero cases collect. A JSON ledger is written totest-runs/and a flock lease stops two runs interleaving.test/parallel-safe.testsandtest/serial.tests: the manifests. Each serial entry says what the global thing is.test/all: delegates to the runner. Exit 0 all passed, 1 any failure (fail-fast, as before), 2 the runner refused to start..github/workflows/test.yml: runstest/allin anarchlinux:latestcontainer on push and pull request. The tests reach forparted, util-linuxscript,bsdtarandlocalectlat the versions that ship on the ISO, which Ubuntu does not have.Testing
test_run_manifest.py: 13 tests covering the process-group timeout, SIGTERM-resistant children, sibling cancellation, leaked background processes, ledger shape, lease contention, empty python modules, and the manifest completeness rules.test/allon Linux is what the new workflow runs; on macOS two upstream tests fail for lack oflocalectlandscript -c, unchanged from before.This is the first of two ISO PRs; the aarch64 ISO PR stacks on it because its new tests need manifest entries.