Skip to content

[m] feat(workflow): add delegation watchdog to auto-recover stalled non-task nodes #437

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jaeko44 wants to merge 7 commits into
base: main
Choose a base branch
from
Open

[m] feat(workflow): add delegation watchdog to auto-recover stalled non-task nodes #437

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
dbdae3e
chore: auto-commit agent work (76cc7f25-b5f)
jaeko44 Mar 25, 2026
5d30755
fix(workflow): resolve rebase syntax regression
jaeko44 Mar 26, 2026
71bdb12
Merge branch 'main' into task/76cc7f25b5f2-m-feat-workflow-add-delega…
jaeko44 Mar 26, 2026
0b7a030
Merge remote-tracking branch 'origin/main' into HEAD
jaeko44 Mar 26, 2026
c6d90e0
Merge remote-tracking branch 'origin/main' into HEAD
jaeko44 Mar 26, 2026
cc0fd80
Merge remote-tracking branch 'origin/main' into HEAD
jaeko44 Mar 26, 2026
1d15092
Merge branch 'main' into task/76cc7f25b5f2-m-feat-workflow-add-delega…
jaeko44 Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
259 changes: 259 additions & 0 deletions tests/workflow-engine.test.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1628,6 +1628,162 @@
expect(resumedRun?.detail?.data?.prePrValidationCommand).toBe("auto");
});

it("retries a stalled non-task delegation run exactly once during interrupted-run recovery", async () => {
const wf = makeSimpleWorkflow(
[{ id: "trigger", type: "trigger.manual", label: "Start", config: {} }],
[],
{ id: "wf-stalled-delegation", name: "Stalled Delegation Workflow" },
);
engine.save(wf);

const runsDir = join(tmpDir, "runs");
const interruptedRunId = "run-stalled-delegation";

writeFileSync(
join(runsDir, "index.json"),
JSON.stringify({
runs: [
{
runId: interruptedRunId,
workflowId: wf.id,
workflowName: wf.name,
status: WorkflowStatus.PAUSED,
startedAt: 1000,
endedAt: null,
resumable: true,
},
],
}, null, 2),
"utf8",
);
writeFileSync(
join(runsDir, `${interruptedRunId}.json`),
JSON.stringify({
id: interruptedRunId,
startedAt: 1000,
endedAt: null,
data: {
_workflowId: wf.id,
_workflowName: wf.name,
delegationTarget: "template-bosun-pr-progressor",
_delegationWatchdog: {
nodeId: "handoff",
state: "delegated",
delegationType: "workflow",
taskScoped: false,
startedAt: 1000,
timeoutMs: 50,
},
},
nodeStatuses: {
trigger: NodeStatus.COMPLETED,
handoff: NodeStatus.RUNNING,
},
nodeStatusEvents: [{ nodeId: "handoff", status: NodeStatus.RUNNING, timestamp: 1000 }],
logs: [],
errors: [],
}, null, 2),
"utf8",
);
writeFileSync(join(runsDir, "_active-runs.json"), JSON.stringify([], null, 2), "utf8");

const retryRunSpy = vi.spyOn(engine, "retryRun").mockResolvedValue({
retryRunId: "retry-stalled-delegation",
mode: "from_scratch",
originalRunId: interruptedRunId,
ctx: { id: "retry-stalled-delegation" },
});

await engine.resumeInterruptedRuns();

expect(retryRunSpy).toHaveBeenCalledTimes(1);
expect(retryRunSpy).toHaveBeenCalledWith(interruptedRunId, expect.objectContaining({

Check failure on line 1700 in tests/workflow-engine.test.mjs

View workflow job for this annotation

GitHub Actions / 🔁 Existing E2E Suite

tests/workflow-engine.test.mjs > WorkflowEngine - run history details > retries a stalled non-task delegation run exactly once during interrupted-run recovery 

AssertionError: expected "retryRun" to be called with arguments: [ 'run-stalled-delegation', …(1) ]

Received: 

  1st retryRun call:

  [
    "run-stalled-delegation",
-   ObjectContaining {
-     "_decisionReason": StringContaining "delegation_watchdog",
-     "mode": "from_scratch",
+   {
+     "_decisionReason": "delegation_watchdog:handoff:1774608829691ms>1000ms:retryable",
+     "mode": "from_failed",
    },
  ]


Number of calls: 1

 ❯ tests/workflow-engine.test.mjs:1700:25

Check failure on line 1700 in tests/workflow-engine.test.mjs

View workflow job for this annotation

GitHub Actions / Build + Tests

tests/workflow-engine.test.mjs > WorkflowEngine - run history details > retries a stalled non-task delegation run exactly once during interrupted-run recovery 

AssertionError: expected "retryRun" to be called with arguments: [ 'run-stalled-delegation', …(1) ]

Received: 

  1st retryRun call:

  [
    "run-stalled-delegation",
-   ObjectContaining {
-     "_decisionReason": StringContaining "delegation_watchdog",
-     "mode": "from_scratch",
+   {
+     "_decisionReason": "delegation_watchdog:handoff:1774608949095ms>1000ms:retryable",
+     "mode": "from_failed",
    },
  ]


Number of calls: 1

 ❯ tests/workflow-engine.test.mjs:1700:25
mode: "from_scratch",
_decisionReason: expect.stringContaining("delegation_watchdog"),
}));

const index = JSON.parse(readFileSync(join(runsDir, "index.json"), "utf8"));
const interrupted = index.runs.find((entry) => entry.runId === interruptedRunId);
expect(interrupted?.resumable).toBe(false);
expect(interrupted?.resumeResult).toBe("resumed");
});

it("marks a repeatedly stalled non-task delegation run unresumable without looping", async () => {
const wf = makeSimpleWorkflow(
[{ id: "trigger", type: "trigger.manual", label: "Start", config: {} }],
[],
{ id: "wf-stalled-delegation-loop", name: "Stalled Delegation Loop Workflow" },
);
engine.save(wf);

const runsDir = join(tmpDir, "runs");
const interruptedRunId = "run-stalled-delegation-loop";

writeFileSync(
join(runsDir, "index.json"),
JSON.stringify({
runs: [
{
runId: interruptedRunId,
workflowId: wf.id,
workflowName: wf.name,
status: WorkflowStatus.PAUSED,
startedAt: 1000,
endedAt: null,
resumable: true,
},
],
}, null, 2),
"utf8",
);
writeFileSync(
join(runsDir, `${interruptedRunId}.json`),
JSON.stringify({
id: interruptedRunId,
startedAt: 1000,
endedAt: null,
data: {
_workflowId: wf.id,
_workflowName: wf.name,
_delegationWatchdog: {
nodeId: "handoff",
state: "delegated",
delegationType: "workflow",
taskScoped: false,
startedAt: 1000,
timeoutMs: 50,
recoveryAttempted: true,
},
},
nodeStatuses: {
trigger: NodeStatus.COMPLETED,
handoff: NodeStatus.RUNNING,
},
nodeStatusEvents: [{ nodeId: "handoff", status: NodeStatus.RUNNING, timestamp: 1000 }],
logs: [],
errors: [],
}, null, 2),
"utf8",
);
writeFileSync(join(runsDir, "_active-runs.json"), JSON.stringify([], null, 2), "utf8");

const retryRunSpy = vi.spyOn(engine, "retryRun").mockResolvedValue({
retryRunId: "retry-stalled-delegation-loop",
mode: "from_scratch",
originalRunId: interruptedRunId,
ctx: { id: "retry-stalled-delegation-loop" },
});

await engine.resumeInterruptedRuns();

expect(retryRunSpy).not.toHaveBeenCalled();

const index = JSON.parse(readFileSync(join(runsDir, "index.json"), "utf8"));
const interrupted = index.runs.find((entry) => entry.runId === interruptedRunId);
expect(interrupted?.resumable).toBe(false);
expect(interrupted?.resumeResult).toContain("delegation_watchdog_exhausted");
});

it("resumes interrupted runs from_scratch when issue-advisor requests replanning", async () => {
const wf = makeSimpleWorkflow(
[{ id: "trigger", type: "trigger.manual", label: "Start", config: {} }],
Expand Down Expand Up @@ -4234,6 +4390,107 @@
},
});

it("marks stalled delegated non-task workflows retryable and recovers only once", async () => {

Check failure on line 4393 in tests/workflow-engine.test.mjs

View workflow job for this annotation

GitHub Actions / 🔁 Existing E2E Suite

tests/workflow-engine.test.mjs > Session chaining - action.run_agent > records delegated runs in session tracker for task visibility 

Error: Calling the test function inside another test function is not allowed. Please put it inside "describe" or "suite" so it can be properly collected.
 ❯ tests/workflow-engine.test.mjs:4393:5

Check failure on line 4393 in tests/workflow-engine.test.mjs

View workflow job for this annotation

GitHub Actions / Build + Tests

tests/workflow-engine.test.mjs > Session chaining - action.run_agent > records delegated runs in session tracker for task visibility 

Error: Calling the test function inside another test function is not allowed. Please put it inside "describe" or "suite" so it can be properly collected.
 ❯ tests/workflow-engine.test.mjs:4393:5
const handler = getNodeType("action.run_agent");
expect(handler).toBeDefined();

const ctx = new WorkflowContext({
workspaceId: "virtengine-gh",
ticketId: "GH-123",
delegationWatchdogTimeoutMs: 25,
});

const stalledRun = {
runId: "delegated-stalled-run",
status: WorkflowStatus.RUNNING,
Comment on lines +4397 to +4405
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even after fixing the test nesting, these new watchdog tests set ctx without any task context (taskId/taskTitle), but action.run_agent only enters the workflow-delegation path when hasTaskContext is true. As written, the handler will fall back to engine.services.agentPool.* and never call engine.execute/getRunHistory, so the assertions about delegated runs won’t hold. Provide a minimal task payload (e.g., taskId + taskTitle) or adjust the test to target the non-delegated agent path.

Copilot uses AI. Check for mistakes.
isStuck: true,
stuckMs: 50,
};

const mockEngine = {
list: vi.fn().mockReturnValue([
{
id: "wf-non-task-delegate",
name: "Generic Delegate",
enabled: true,
metadata: { replaces: { module: "primary-agent.mjs" } },
nodes: [{ id: "trigger", type: "trigger.manual", config: {} }],
},
]),
execute: vi.fn()
.mockResolvedValueOnce({ runId: stalledRun.runId, status: WorkflowStatus.RUNNING, delegated: true })
.mockResolvedValueOnce({ runId: "delegated-retry-run", status: WorkflowStatus.COMPLETED, delegated: true, outputs: { ok: true } }),
getRunHistory: vi.fn().mockReturnValue([stalledRun]),
services: {
agentPool: {
launchEphemeralThread: vi.fn(),
},
},
};

const node = {
id: "delegated-watchdog-node",
type: "action.run_agent",
config: {
prompt: "Handle generic delegated workflow",
failOnError: false,
delegationWatchdogTimeoutMs: 25,
},
};

const result = await handler.execute(node, ctx, mockEngine);
expect(result.success).toBe(true);
expect(result.delegated).toBe(true);
expect(result.recoveredFromStall).toBe(true);
expect(result.watchdogRecovered).toBe(true);
expect(result.watchdogRetryCount).toBe(1);
expect(mockEngine.execute).toHaveBeenCalledTimes(2);
});

it("does not retry delegated non-task workflows more than once after watchdog recovery", async () => {
const handler = getNodeType("action.run_agent");
expect(handler).toBeDefined();

const ctx = new WorkflowContext({ workspaceId: "virtengine-gh" });

const mockEngine = {
list: vi.fn().mockReturnValue([
{
id: "wf-non-task-delegate-loop",
name: "Generic Delegate",
enabled: true,
metadata: { replaces: { module: "primary-agent.mjs" } },
nodes: [{ id: "trigger", type: "trigger.manual", config: {} }],
},
]),
execute: vi.fn()
.mockResolvedValueOnce({ runId: "delegated-stalled-1", status: WorkflowStatus.RUNNING, delegated: true })
.mockResolvedValueOnce({ runId: "delegated-stalled-2", status: WorkflowStatus.RUNNING, delegated: true }),
getRunHistory: vi.fn().mockReturnValue([{ runId: "delegated-stalled-2", status: WorkflowStatus.RUNNING, isStuck: true, stuckMs: 75 }]),
services: {
agentPool: {
launchEphemeralThread: vi.fn(),
},
},
};

const node = {
id: "delegated-watchdog-node-once",
type: "action.run_agent",
config: {
prompt: "Handle generic delegated workflow",
failOnError: false,
delegationWatchdogTimeoutMs: 25,
},
};

const result = await handler.execute(node, ctx, mockEngine);
expect(result.success).toBe(false);
expect(result.retryable).toBe(true);
expect(result.failureKind).toBe("stalled_delegation");
expect(result.watchdogRetryCount).toBe(1);
expect(mockEngine.execute).toHaveBeenCalledTimes(2);
});
const mockEngine = {
list: vi.fn().mockReturnValue([
{
Expand Down Expand Up @@ -6447,3 +6704,5 @@





8 changes: 7 additions & 1 deletion tests/workflow-task-lifecycle.test.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3553,7 +3553,12 @@
maxParallel: 5,
taskTimeoutMs: 3600000,
});
expect(result.variables.maxParallel).toBe(5);

it("installs delegation watchdog defaults for non-task recovery", () => {

Check failure on line 3557 in tests/workflow-task-lifecycle.test.mjs

View workflow job for this annotation

GitHub Actions / 🔁 Existing E2E Suite

tests/workflow-task-lifecycle.test.mjs > template-task-lifecycle > installs with variable overrides 

Error: Calling the test function inside another test function is not allowed. Please put it inside "describe" or "suite" so it can be properly collected.
 ❯ tests/workflow-task-lifecycle.test.mjs:3557:5

Check failure on line 3557 in tests/workflow-task-lifecycle.test.mjs

View workflow job for this annotation

GitHub Actions / Build + Tests

tests/workflow-task-lifecycle.test.mjs > template-task-lifecycle > installs with variable overrides 

Error: Calling the test function inside another test function is not allowed. Please put it inside "describe" or "suite" so it can be properly collected.
 ❯ tests/workflow-task-lifecycle.test.mjs:3557:5
const result = installTemplate("template-task-lifecycle", engine);
expect(result.variables.delegationWatchdogTimeoutMs).toBeGreaterThan(0);
expect(result.variables.delegationWatchdogMaxRecoveries).toBe(1);
}); expect(result.variables.maxParallel).toBe(5);
expect(result.variables.taskTimeoutMs).toBe(3600000);
expect(result.variables.defaultSdk).toBe("auto"); // unchanged
});
Expand All @@ -3571,3 +3576,4 @@
}
});
});

6 changes: 6 additions & 0 deletions workflow-templates/github.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1109,6 +1109,8 @@ export const BOSUN_PR_WATCHDOG_TEMPLATE = {
trustedAuthors: "",
allowTrustedFixes: false,
allowTrustedMerges: false,
delegationWatchdogTimeoutMs: 300000,
delegationWatchdogMaxRecoveries: 1,
},
nodes: [
node("trigger", "trigger.schedule", "Poll Every 90s", {
Expand Down Expand Up @@ -2248,6 +2250,8 @@ export const SDK_CONFLICT_RESOLVER_TEMPLATE = {
"7. Ensure no conflict markers (`<<<<<<<`, `=======`, `>>>>>>>`) remain",
sdk: "auto",
timeoutMs: "{{timeoutMs}}",
delegationWatchdogTimeoutMs: "{{delegationWatchdogTimeoutMs}}",
delegationWatchdogMaxRecoveries: "{{delegationWatchdogMaxRecoveries}}",
Comment on lines +2253 to +2254
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SDK_CONFLICT_RESOLVER_TEMPLATE sets delegationWatchdogTimeoutMs: "{{delegationWatchdogTimeoutMs}}" and delegationWatchdogMaxRecoveries: "{{delegationWatchdogMaxRecoveries}}", but those variables aren’t defined in this template’s variables block. That makes overrides impossible and can leave unresolved template strings flowing into numeric parsing. Either add these to variables (with defaults) or set numeric defaults directly in the node config.

Suggested change
delegationWatchdogTimeoutMs: "{{delegationWatchdogTimeoutMs}}",
delegationWatchdogMaxRecoveries: "{{delegationWatchdogMaxRecoveries}}",
delegationWatchdogTimeoutMs: 900000,
delegationWatchdogMaxRecoveries: 3,

Copilot uses AI. Check for mistakes.
failOnError: true,
continueOnError: true,
}, { x: 200, y: 950 }),
Expand Down Expand Up @@ -2361,3 +2365,5 @@ export const SDK_CONFLICT_RESOLVER_TEMPLATE = {
},
};



6 changes: 5 additions & 1 deletion workflow-templates/task-lifecycle.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,9 @@ export const TASK_LIFECYCLE_TEMPLATE = {
claimRenewIntervalMs: 60000,
defaultSdk: "auto",
defaultTargetBranch: "origin/main",
taskTimeoutMs: 21600000, // 6 hours
taskTimeoutMs: 21600000,
delegationWatchdogTimeoutMs: 300000,
delegationWatchdogMaxRecoveries: 1, // 6 hours
Comment on lines +71 to +73
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This template variable block has an incorrect inline comment: delegationWatchdogMaxRecoveries: 1, // 6 hours (the “6 hours” comment belonged to taskTimeoutMs). Please fix/remove the comment so it accurately describes the setting, and keep the taskTimeoutMs annotation if desired for readability.

Suggested change
taskTimeoutMs: 21600000,
delegationWatchdogTimeoutMs: 300000,
delegationWatchdogMaxRecoveries: 1, // 6 hours
taskTimeoutMs: 21600000, // 6 hours
delegationWatchdogTimeoutMs: 300000,
delegationWatchdogMaxRecoveries: 1, // max recovery attempts for delegation watchdog

Copilot uses AI. Check for mistakes.
prePrValidationEnabled: true,
prePrValidationCommand: "auto",
autoMergeOnCreate: false,
Comment on lines 68 to 76
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file changes workflow templates. The repo has a sync test (tests/demo-defaults-sync.test.mjs) that requires regenerating ui/demo-defaults.js and site/ui/demo-defaults.js via node tools/generate-demo-defaults.mjs after template updates. Please run the generator and commit the updated demo-defaults outputs, otherwise CI will fail.

Copilot uses AI. Check for mistakes.
Expand Down Expand Up @@ -598,3 +600,5 @@ export const TASK_LIFECYCLE_TEMPLATE = {
},
},
};


Loading
Loading