feat(jobs): handle SIGTERM in job runner for graceful shutdown from Docker/systemd

[lisa-assistant]

Background

In packages/jobs/src/bins/rw-jobs.ts, the signalSetup function (line 187) only sets up a handler for SIGINT (Ctrl+C):

process.on('SIGINT', () => {
  // forwards SIGINT to each worker for graceful shutdown
  workers.forEach(worker => worker.kill('SIGINT'))
})

There is a TODO at line 150 noting that SIGTERM/SIGKILL support is missing:

// TODO add support for stopping with SIGTERM or SIGKILL?
export const stopWorkers = async ({ ... signal = 'SIGINT', ... }) => { ... }

SIGTERM is the standard signal sent by Docker, Kubernetes, and systemd when stopping a container or service. Without a SIGTERM handler in the parent process, the job runner ignores graceful shutdown requests from container orchestration — workers may be hard-killed without finishing their current job.

What needs to be done

Add a SIGTERM handler in signalSetup that forwards graceful shutdown to workers, mirroring the SIGINT behavior:

// SIGTERM: sent by Docker, systemd, Kubernetes during shutdown
process.on('SIGTERM', () => {
  logger.info('SIGTERM received: shutting down workers gracefully...')
  workers.forEach((worker) => {
    worker.kill('SIGTERM')
  })
})

This ensures that when the container/service is stopped, in-progress jobs get a chance to complete or be re-queued rather than being abruptly killed.

Additional consideration

The stop command in main() currently always uses SIGINT:

case 'stop':
  return await stopWorkers({ numWorkers, signal: 'SIGINT', logger })

Consider adding a --signal flag to the CLI so operators can choose SIGTERM for deployment scenarios.

File

packages/jobs/src/bins/rw-jobs.ts, lines 150 and 187–220

[lisa-assistant]

Self-fix confidence: 8/10

The core change is straightforward — add a process.on('SIGTERM', ...) handler following the exact pattern already established for SIGINT. The 2-point deduction comes from two areas of uncertainty:

1. Signal semantics: Should SIGTERM trigger the same "press again to force quit" double-tap behavior as SIGINT, or should it be a single-pass graceful shutdown? Systemd typically gives a grace period then sends SIGKILL, so a single-pass forward is probably correct — but this is a design decision.
2. Worker process behavior: The workers themselves also need to handle SIGTERM gracefully (in Worker.ts). If a worker ignores SIGTERM and only handles SIGINT, forwarding SIGTERM from the parent won't help. I'd need to audit the worker's signal handling before this is fully useful.

[lisa-assistant]

Implementation Plan

Context

The parent process () only has a handler in . Workers () already handle both signals:

• SIGINT (line 66): sets , lets the current job finish then exits
• SIGTERM (line 77): exits immediately via

So forwarding SIGTERM from parent → workers is safe and the workers will handle it correctly.

Root cause

When Docker/systemd sends SIGTERM to the parent process, Node's default behavior is to exit immediately without calling any signal handlers and without forwarding to children. The worker child processes become orphaned rather than shut down gracefully.

---

Changes required

1. — add SIGTERM to signalSetup

Design decision: SIGTERM uses a single-pass forward (no double-tap counter). The worker exits immediately on SIGTERM anyway, and the orchestrator (Docker/systemd) handles the kill timeout itself.

2. — fix the stop command to use SIGTERM

The stop command targets already-daemonized workers by PID. SIGTERM is the correct signal for requesting a graceful stop of a daemon:

Note: uses which sends to an external PID, so SIGTERM is semantically correct here. The worker's SIGTERM handler () ensures it exits cleanly.

---

Edge cases

• Detached workers: When workers are started with , won't reach them because the ChildProcess handle is unreferenced. The / path (which uses ) is the correct path for daemon shutdown — and that already works.
• Double SIGTERM: If the orchestrator sends SIGTERM twice, the parent will forward both. Workers will call on the first one; subsequent signals are ignored since the process is already exiting.
• Tests: already guards with . Signal handler tests can check after calling .

---

Files to change

File	Change
	Add handler in ; change / to use

That's it — one file, ~10 lines added/changed.

[lisa-assistant]

Self-fix confidence: 8/10

The worker already handles SIGTERM (immediate exit) — the parent just needs to forward it. The core change is ~5 lines in signalSetup. The 2-point deduction is for the design decision around whether SIGTERM should also forward as SIGINT first (for a graceful job-completion window) vs forwarding SIGTERM directly (immediate worker exit) — and for validating that daemonized workers (detached, unreferenced) are correctly stopped via the existing process.kill(pid) path rather than the worker.kill() path.