Diagnostics: record what holds a shard open after its last test - #566
Merged
Conversation
Shards report every test ok and then sit silent until the job's cap, and nothing about the code has explained it: the guards were made uniform and a shard still hung on a composition that reaches none of them. So this stops guessing and records the state instead. A sampler snapshots ps, the fds of every surviving node and python, and any bats temp remnants, every 30s while the suite runs. Forensics run under always() so they also fire when the job is cancelled at its cap, which is the only case that matters. The sampler writes to a file and never to the step's stdout, and closes fds 3 and 4: a process holding that pipe is one of the suspects, so the instrument must not be able to cause what it measures. It also stops on its own after 23 minutes, and as soon as the suite finishes. Both runners are instrumented, not just macOS, because the few Linux occurrences are the closest thing to a control group available.
The forensics step ran under the runner's default `bash -e`, so an lsof against a pid that had just exited, or a grep matching nothing, could have turned a green shard red -- an instrument breaking the build it was added to measure. All three diagnostic steps are continue-on-error now, the forensics body drops -e and pipefail and ends in exit 0, and head is gone from the pipelines because it SIGPIPEs whatever is upstream. Only the bats step decides the shard's result, which is what it was before any of this was added. Checked by extracting the step from the YAML as the runner receives it and running it under both `bash -e` and `bash -eo pipefail`: exit 0 both times, with every section still produced.
Dropping head from the pipelines was meant to stop it SIGPIPEing what was upstream. It also narrowed the search: `ls | grep -i bats | head -20` matched case-insensitively across every entry, while the replacement matched case-sensitively inside the first 400 lines of the listing. On a temp dir with more than 400 entries, or a remnant whose name differs in case, the probe would have reported nothing while the thing it looks for sat there. Matching with tolower() over the whole listing and counting matches rather than lines restores the original contract and still needs no pipe to cut short. Same for the sampler's copy of that probe and its per-pid fd listing. Checked against a directory holding 450 fillers plus three remnants placed exactly where the narrowed version would miss them: the original finds 3, the narrowed version finds 0, this finds 3.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Instrumentation only. No change to the suite's behaviour.
Shards report every test
okand then sit silent until the job's 25-minute cap. Three explanations have been tried and each was refuted by observation — most recently making the fd-3 spawn guard uniform, after which a shard still hung on a composition that reaches none of the guarded spawns. Rather than propose a fourth, this records the state while the hang is live.Sampler
Every 30s during the suite:
ps -ef, the open fds of every survivingnode/python(what each holds, and whether it is a pipe), and anybats-run-*temp remnants.It writes to a file and never to the step's stdout, and it closes fds 3 and 4. A process holding that pipe is one of the suspects, so the instrument must not be able to cause the thing it is measuring. It stops when the suite finishes, and independently after 23 minutes, so it cannot outlive the job it is watching.
Forensics
Under
always(), so they run when the job is cancelled at its cap — the only case that matters. A live snapshot, then everything still holding a pipe or fifo with its command line, then the sampler timeline. Also uploaded as an artifact, because a cancelled job can cut its log short.Both runners
Linux hangs are rarer than macOS ones, which makes them the closest thing to a control group available. Instrumenting only the failing platform would leave nothing to compare against.
Verification
The sampler step was extracted from the YAML exactly as the runner receives it and executed: all three sections are produced, including the fd listings, and the sentinel stops it. The workflow parses, and the repo's own
test_ci_workflow/test_ci_shardingsuites pass.What this cannot do
If the shard composition on this branch does not happen to hang, this captures nothing. Whether a given shard hangs has tracked the branch's file partition rather than luck, so the fallback is to carry these steps onto a branch whose composition is known to hang.