Context
While debugging why an externally-cancelled Slurm job never tripped a client's job_status failure watch (PR #102, commit 2b66eca), the root cause turned out to be an upstream API quirk: the SFAPI single-job status route chronically returned empty output, and the poller — which deliberately tolerates empty output for the post-submit sacct lag — treated permanent blindness as "no change", forever. No notification was ever emitted for any job; nothing logged; the polling cadence looked perfectly healthy in the broker log.
2b66eca fixed that instance (query via the list route + cached=false, the way NERSC's own sfapi_client does). This ticket is about the class:
Problem
A start_status_poller-based poller (sfapi and iri instance plugins) has no way to distinguish "job state hasn't changed" from "I have been receiving zero information for an hour". Consumers (e.g. a client-side failure watch waiting on terminal states) then block until their own timeout — 30 min in the amsc launch driver.
Proposal
- Track consecutive no-data polls per job in the shared poller (
plugin_session_base.start_status_poller). After a threshold (e.g. N polls ≈ 5 minutes) for a job that has previously been seen (or since submit for one that never appeared):
- log a WARNING naming the job and the route, and
- optionally emit a synthetic
job_status notification with state: 'unknown' so downstream watchers can react instead of hanging.
- Commit a transport-level test for hosted-plugin notification delivery (broker
_host_broadcast → EventRouter → subscribed WS participant). This path had no coverage; a working reproduction exists from the debugging session and can be adapted.
References
🤖 Generated with Claude Code
Context
While debugging why an externally-cancelled Slurm job never tripped a client's
job_statusfailure watch (PR #102, commit 2b66eca), the root cause turned out to be an upstream API quirk: the SFAPI single-job status route chronically returned emptyoutput, and the poller — which deliberately tolerates empty output for the post-submit sacct lag — treated permanent blindness as "no change", forever. No notification was ever emitted for any job; nothing logged; the polling cadence looked perfectly healthy in the broker log.2b66eca fixed that instance (query via the list route +
cached=false, the way NERSC's ownsfapi_clientdoes). This ticket is about the class:Problem
A
start_status_poller-based poller (sfapi and iri instance plugins) has no way to distinguish "job state hasn't changed" from "I have been receiving zero information for an hour". Consumers (e.g. a client-side failure watch waiting on terminal states) then block until their own timeout — 30 min in the amsc launch driver.Proposal
plugin_session_base.start_status_poller). After a threshold (e.g. N polls ≈ 5 minutes) for a job that has previously been seen (or since submit for one that never appeared):job_statusnotification withstate: 'unknown'so downstream watchers can react instead of hanging._host_broadcast→ EventRouter → subscribed WS participant). This path had no coverage; a working reproduction exists from the debugging session and can be adapted.References
feature/sfapi), commit 2b66eca — instance fix + route regression testsGET /compute/jobs/{machine}/{jobid}?sacct=truewith the defaultcached=truereturns emptyoutputindefinitely;sfapi_clientusesGET /compute/jobs/{machine}?sacct=true&cached=false&kwargs=jobid=Ninstead (being reported to NERSC separately).🤖 Generated with Claude Code