Bound background task finalization so one failed task cannot starve index builds

[zcourts]

Problem

A failed background task can retain the worker semaphore permit indefinitely while recording its failure. On a single-shard topology Anvil currently forces background concurrency to one, so this stalls the entire queue and leaves later typed indexes enabled but unmaterialized.

Worka E2E34 evidence (2026-07-30)

Candidate source: a6b8f719f36190800dcdcceb0c06957270db84bf.

• Task 110 (AuthzMaterialization) failed at 05:48:25 on an authz-head ValueHash predicate.
• No failure-recording result, task completion, or later index-build completion was logged.
• At 05:55:41 recovery found task 110 still running without a valid execution lease and requeued it.
• All bootstrap index builds had completed by 05:48:29, but none of 11 indexes created for a template workspace around 06:07 materialized.
• workspace.integration_events_by_time was present, enabled, and version 1, but had no typed segment.
• Worka queried it with lag_timeout_ms=5000; TypedJsonIndexNotMaterialised stopped the watcher every 15 seconds despite caller reconciliation.

The worker error branch awaits execution_lease_precondition and then fail_task_with_execution_guard inside the spawned task while holding its semaphore permit. Neither await has an outer deadline. Recovery can requeue the task record but cannot release a permit retained by the stuck spawned future.

Single-node serialization compounds the defect: effective_worker_concurrency returns 1 whenever fewer than two shard targets exist, even though durability topology should not serialize unrelated background task execution.

Acceptance criteria

• Failure and completion persistence have bounded deadlines and cannot retain a worker permit indefinitely.
• Cancellation leaves a recoverable queue/lease state; recovery can safely reclaim the exact attempt.
• A deliberately hung failure-finalization operation does not prevent an unrelated queued index build from starting.
• A failed authz materialization does not starve later index builds.
• Background concurrency is not reduced to one solely because local durability has one shard target; configured concurrency remains bounded and MVCC/task leases remain authoritative.
• Focused tests cover timeout, permit release, recovery, and single-node concurrency.
• Worka's newly created workspace indexes materialize before/within their query lag timeout under the E2E workload.

[zcourts]

Source fixes pushed in two logical commits (no build/test run while Worka E2E34 is live):

• d358b03 decouples configured background concurrency from shard/durability topology and updates the focused unit expectation.
• 3dd647b bounds failure/completion outcome persistence at 30 seconds, guaranteeing the spawned attempt drops its semaphore permit on timeout and leaves lease/attempt recovery authoritative. Added focused async tests for timeout/permit release and normal completion.

Static validation: standalone rustfmt, git diff --check, worker.rs 1491 lines (<2K). Full Cargo tests and image qualification remain intentionally deferred until the live HMO run terminates and the known-fix batch is complete.

[zcourts]

ARM fix2 image qualification (Worka HMO E2E35 gate)

Built from worka/v0.1.0-anvil-rc9 at 2c4fef6831de6779db66ecff3f3c17d9c1181053:

• image: worka/anvil:v0.4.1-worka-v0.1.0-rc9-fix2-arm64
• image ID: sha256:e8fe1828befd334e6e4cdda8ae5c5dbb11a782e15448b95b6a5cf82f037ccde0
• platform: linux/arm64
• release build completed successfully using the shared Cargo target with one job
• staged Docker binaries exactly match the release binary SHA-256 digests
• anvil-server --version, anvil --version, and anvil-admin --version pass in the built image
• source branch is clean and exactly synced with origin

The image has been handed to the Worka agent for E2E35. End-to-end qualification remains pending; this comment does not claim the issue is closed.

[zcourts]

E2E35 live qualification update

The fresh Worka runtime using worka/anvil:v0.4.1-worka-v0.1.0-rc9-fix2-arm64 is showing continuous Index build task completed events across multiple task IDs while object materialisation and persistence bootstrap proceed. The E2E34 worker-halt / TypedJsonIndexNotMaterialised starvation signature has not recurred so far.

This is positive live evidence for the worker-concurrency and bounded-finalization changes, but the end-to-end run is still in progress, so the issue remains open.

[zcourts]

Fresh Worka E2E35 against the consolidated fix2 candidate shows #236 is not yet resolved and remains a Worka v0.1.0 release blocker.

Candidate and preserved environment:

• Image: worka/anvil:v0.4.1-worka-v0.1.0-rc9-fix2-arm64
• Image ID: sha256:e8fe1828befd334e6e4cdda8ae5c5dbb11a782e15448b95b6a5cf82f037ccde0
• Source: 2c4fef6831de6779db66ecff3f3c17d9c1181053
• Container: release-anvil-v041-e2e-35-arm64 (healthy, port 55101)
• Volumes: release_anvil-v041-e2e35-data, release_anvil-v041-e2e35-bootstrap-data
• Worka logs: .tmp/worka/hmo-e2e35-fix2.log, .tmp/worka/cloud-api.log, .tmp/worka/fission-live-headless-client.log

Exact timeline:

• 08:50:59–08:52:40: AuthzMaterialization task IDs 1, 3, 7, 11, 15, 17, 19 and 23 fail with MVCC value-hash/predicate conflicts.
• 08:51:01–08:54:36: other workers continue completing typed-JSON index tasks, initially demonstrating that increased concurrency prevents one failure from immediately monopolising the queue.
• 08:54:36: final observed Index build task completed (index ID 35, generation/source cursor 70647809).
• 08:54:36 onward: zero further index completions despite continued Worka activity.
• 09:34:02 onward: Worka template-processor maintenance and build-recovery passes repeatedly receive TypedJsonIndexNotMaterialised.
• ~09:36: the first synchronous template indexing command, worka-admin marketplace trust github-org worka-ai, fails with TypedJsonIndexNotMaterialised; E2E35 exits at scripts/index_worka_templates_local.sh line 943.

The run used entirely fresh Anvil state. Prior boundaries succeeded: Worka persistence initialization, publisher CAS, GreenMail offer, email-pack publication, headless/controller and build-service registration. The failure happened before template publication/clone.

Conclusion: raising worker concurrency delayed the starvation observed in E2E34 but did not fix the underlying lease/queue lifecycle after repeated AuthzMaterialization predicate conflicts. Worka will not add retries or weaken indexed-query semantics. The preserved container and volumes should be inspected in place before another candidate build; no Worka rerun is planned until the root fix is ready.

[zcourts]

E2E35 root-cause follow-up and fix3 qualification

E2E35 showed that increased worker concurrency delayed but did not eliminate index starvation. The preserved timeline established the exact scheduler failure shape: already-running index tasks drained after the last authorization-materialization conflict, then all worker progress stopped because index/bucket/tenant finalization passes were unbounded inline awaits ahead of task claims.

Commit 0cd6580031ec6b9e7381d003983e97c974aaff38 moves each maintenance kind into a non-overlapping single-flight execution owned by the worker. The claim loop polls pass completion without awaiting a pending pass, logs pass age/stalls, and continues claiming unrelated due tasks. Maintenance is not timeout-cancelled and same-kind passes cannot overlap.

Focused regression:

• worker::tests::stalled_maintenance_pass_remains_single_flight_and_does_not_block_task_claims
• result: 1 passed, 0 failed
• proves a never-returning maintenance pass starts exactly once while a real unrelated IndexBuild task is still claimed

Qualified ARM image:

• worka/anvil:v0.4.1-worka-v0.1.0-rc9-fix3-arm64
• ID sha256:d361ec6a31772fbff47b79a5a6cb84ab9057f54dd1878ce50ef724b51dc1cd7d
• linux/arm64
• exact source 0cd6580031ec6b9e7381d003983e97c974aaff38
• release build, source identity/cleanliness, staged-binary checksum equality, and all three container CLI smoke gates passed

Worka E2E36 remains the end-to-end qualification gate; this comment does not close the issue.

[zcourts]

fix3 E2E36 terminal reproduction — ARM64

Result: fix3 improves bootstrap progress but still fails the Worka HMO release path because typed-json index finalization/availability does not converge. No retry or build was started.

• Image: worka/anvil:v0.4.1-worka-v0.1.0-rc9-fix3-arm64
• Exact image ID: sha256:d361ec6a31772fbff47b79a5a6cb84ab9057f54dd1878ce50ef724b51dc1cd7d
• Source: 0cd6580
• Fresh isolated container/port: release-anvil-v041-e2e-36-arm64, 55102
• Fresh dedicated volumes preserved: release_anvil-v041-e2e36-data, release_anvil-v041-e2e36-bootstrap-data

Progress compared with the prior starvation

During Worka bootstrap, authz/materialisation conflicts occurred but typed-json task claims and completions continued. Maintenance warnings correctly reported that claims continued while a finalization pass was in flight. Worka reached system_agents at 13:47:41 UTC and cleared it at 13:56:14; all remaining persistence initialization completed and the API substrate was ready at 13:56:17.

Final observed typed-json completions were:

• index 56 at 13:55:41
• index 55 at 13:55:52
• index 37 at 13:56:05
• index 32 at 13:56:13

Index 27 was skipped once as absent/disabled/stale/unsupported. No individual index/task ID was observed repeatedly pending.

Remaining failure

After index 32, no further index completion appeared. One index_finalization pass remained continuously in flight, with warning ages growing through 40.7s, 51.1s, 61.3s, 71.8s, 81.9s, 92.3s, 102.6s, 112.9s and 123.4s. Object materialisation first exceeded its 29-second lease-safe timeout, then repeatedly failed with transaction is no longer open (13:56:58, 13:57:16, 13:57:32, 13:57:55, 13:58:08).

The Worka template processor received TypedJsonIndexNotMaterialised on every subsequent pass at 13:56:29, 13:56:46, 13:57:02, 13:57:19, 13:57:37, 13:57:54, 13:58:12, and 13:58:30.

At 13:58:35, Worka publisher ProvisionDevice failed because its required auth query returned TypedJsonIndexNotMaterialised; the RPC surfaced as Internal: authentication service operation failed. This happened after the publisher tenant was created but before any pack publication.

Release impact

This remains release-blocking for Worka 0.1.0. The HMO E2E cannot reach email pack publication, template indexing, GreenMail, elicitation, workspace assertions, or UI evidence. Fix3 solves the earlier absolute claim starvation but not the stale/non-materialised index state caused by the stuck finalization/closed transaction.

All E2E36 evidence and the live healthy Anvil container/volumes are preserved for diagnosis. The run used pinned runtime binaries and invoked no Cargo/rustc.