fix(deployments): handle Docker restart races

[ironcommit]

Summary

Docker deployment status reads now reconcile from Docker inspect state instead of mutating containers in the read path. This preserves one-shot and retry semantics across controller restarts and avoids releasing Docker GPU allocations before a deployment reaches a terminal status.

This PR no longer carries the Authentik/Envoy keep-alive template changes; that work has already merged into main via #1101.

Changes

• Read Docker State.ExitCode and RestartCount from inspect attrs for create-time status mapping and read_status().
• Keep stopped Never containers observable and report their terminal status without removing them from read_status().
• Preserve retryable OnFailure deployments by comparing Docker's persisted restart count with the deployment backoff limit.
• Require positive backoffLimit / backoff_limit values in DeploymentConfig, the create request schema, and generated OpenAPI.
• Guard Docker on-failure run kwargs against MaximumRetryCount=0, and treat legacy zero-backoff Docker labels as unlimited retry and non-terminal so GPU allocations are not released while Docker would keep retrying.
• Release Docker GPU allocations only when status handling reaches a terminal deployment result.
• Map missing containers through restart-policy-aware status so Always stays recoverable as LOST while one-shot policies report terminal failure when the backend resource is gone.
• Treat foreign resource-scope containers like missing containers, including GPU release for terminal one-shot status results.
• Add Docker unit and integration coverage for terminal containers, retryable and exhausted OnFailure exits, missing one-shot containers, lost Always containers, foreign-container mismatch handling, zero-backoff legacy handling, API/entity validation, and GPU-release behavior.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with documentation updates
• Documentation only
• Contributor tooling or automation
• CI, build, or test infrastructure

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Documentation updated for user-visible behavior
• Documentation not applicable — justification:

Verification

• Pull request title follows the repository's Conventional Commit format
• Every commit includes an appropriate Signed-off-by: trailer
• uv run pre-commit run -a passes, or any blocked checks are identified below
• Targeted tests pass, or tests are marked not applicable above
• No secrets, API keys, or credentials are included

Targeted validation:

• script/generate-openapi-spec.sh — completed successfully.
• uv run --frozen pytest plugins/nemo-deployments/tests/unit/test_entities.py plugins/nemo-deployments/tests/unit/test_api_deployment_configs.py plugins/nemo-deployments/tests/unit/backends/docker/test_restart_policy.py plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py plugins/nemo-deployments/tests/integration/backends/docker/test_docker_backend.py plugins/nemo-deployments/tests/unit/backends/k8s/test_jobs.py -v — 90 passed.
• uv run pre-commit run -a — blocked by local environment after all other hooks passed. Passed: ruff (legacy alias), ruff format, Run ty typechecks, Check config reference doc is up to date, Helm Docs Container, Run uv lock with platform uv, Check for uv.lock drift, Fix copyright headers, Plugins must not import from nmp-common, and check for merge conflicts. Blocked: studio-lint-staged failed because lint-staged is not installed.
• git diff --cached --check — passed before amend.
• git diff --check origin/main...HEAD — passed after amend.
• DCO audit across origin/main..HEAD — passed for 8b6ad17a7.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Docker status handling now safely reads container exit and restart data, conditionally releases GPUs for terminal missing-container states, and covers Never and OnFailure outcomes with unit and integration tests.

Changes

Docker status handling

Layer / File(s)	Summary
Docker inspection access plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py, plugins/nemo-deployments/tests/unit/backends/docker/docker_helpers.py	Docker imports are moved to module scope, inspection helpers read exit and restart values safely, and test container attributes accept configurable restart counts.
Missing-container and retry status transitions plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py	Missing-container results now determine whether GPU allocations are released, while exited and dead containers use helper-based exit and restart data.
Status behavior validation plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py, plugins/nemo-deployments/tests/integration/backends/docker/test_docker_backend.py	Tests cover missing containers, successful exits, OnFailure retries, backoff-limit failures, GPU release, and shared entity mocking.

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#925: Updates related Docker backend exited-container status mapping.

Suggested labels: test

Suggested reviewers: tylersbray, benmccown

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 8.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: fixing Docker deployment status races during container restarts.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch docker-never-status-race/rsadler

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py (1)

124-188: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

No test coverage for restart_policy="OnFailure" removed-container recovery.

The new recovery logic in backend.py triggers for both "Never" and "OnFailure" (Line 290: if restart_policy in ("Never", "OnFailure")), but both test suites only exercise "Never".

• plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py#L124-L188: add a mocked test with restart_policy="OnFailure" and a non-zero recovered exit code to verify the mapped status (and surface the backoff-bypass behavior flagged in backend.py).
• plugins/nemo-deployments/tests/integration/backends/docker/test_docker_backend.py#L119-L162: add an OnFailure-policy variant of the removed-container integration test using a failing command.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py`
around lines 124 - 188, Extend the mocked tests around
test_read_status_recovers_removed_never_container_exit_code and
test_read_status_failed_when_missing_never_has_no_exit_event in
plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py
(lines 124-188) with an OnFailure case using a non-zero recovered Docker
die-event exit code and assertions for its mapped status and exit code. Add the
corresponding OnFailure removed-container integration-test variant using a
failing command in
plugins/nemo-deployments/tests/integration/backends/docker/test_docker_backend.py
(lines 119-162).

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py`:
- Around line 399-418: The _read_removed_container_status method must preserve
OnFailure retry behavior instead of directly using map_exited_status for
non-zero exits. Apply the same restart_count/backoff_limit handling used for
in-place container exits so eligible failures return STARTING and retries
continue, while exhausted retries retain the terminal FAILED status.
- Around line 399-453: The current _latest_removed_container_exit_code method
depends on Docker’s bounded one-hour event replay and can miss recent container
exits. Capture and retain each container’s exit code when NotFound is first
detected during read_status, or maintain an equivalent live per-container
die-event cache, then have _read_removed_container_status use that recorded
value before falling back to missing_container_status; preserve existing status
mapping and recovery metadata.

---

Nitpick comments:
In `@plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py`:
- Around line 124-188: Extend the mocked tests around
test_read_status_recovers_removed_never_container_exit_code and
test_read_status_failed_when_missing_never_has_no_exit_event in
plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py
(lines 124-188) with an OnFailure case using a non-zero recovered Docker
die-event exit code and assertions for its mapped status and exit code. Add the
corresponding OnFailure removed-container integration-test variant using a
failing command in
plugins/nemo-deployments/tests/integration/backends/docker/test_docker_backend.py
(lines 119-162).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2a4007df-5e5c-4fdd-a755-a09492bea6f1

📥 Commits

Reviewing files that changed from the base of the PR and between d206ae9 and affb68f.

📒 Files selected for processing (3)

• plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py
• plugins/nemo-deployments/tests/integration/backends/docker/test_docker_backend.py
• plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	34297/43312	79.2%	64.1%
Integration Tests	20251/41111	49.3%	22.0%

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py`:
- Around line 633-682: Update _read_removed_container_status to accept a dep_key
parameter and release the GPU via self._gpu_pool.release_gpu(dep_key) whenever
recovered exit handling reaches a terminal status, including success and
non-retryable failure, while preserving the retry/STARTING path without release.
Pass deployment_key(workspace, name) from the read_status call site.
- Around line 656-671: Update the OnFailure handling in backend.py around the
restart_policy check to read RestartCount from cached inspect state or another
persisted pre-removal source rather than die-event attrs, preserving the
backoff-limit behavior. In test_backend_mocked.py lines 507-543 and 546-581,
update both unit cases to provide/assert the real persisted restart-count signal
instead of placing restartCount in event attributes.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 098f3f89-e0e0-4944-aa9a-0515eea3c3e6

📥 Commits

Reviewing files that changed from the base of the PR and between affb68f and fd3772a.

📒 Files selected for processing (3)

• plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py
• plugins/nemo-deployments/tests/integration/backends/docker/test_docker_backend.py
• plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py

🚧 Files skipped from review as they are similar to previous changes (1)

• plugins/nemo-deployments/tests/integration/backends/docker/test_docker_backend.py

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-868-docker-never-status-race-rsadler.docs.buildwithfern.com/nemo-platform

[tylersbray]

agent (triple-review): R6 — remaining OnFailure backoff_limit=0 race (pre-existing)

Severity: P3 · Blocking: no
Models: Opus (single-source) · parent-verified as pre-existing (not introduced by this PR’s hunks)

Summary: For OnFailure, Docker MaximumRetryCount: 0 means unlimited retries (restart_policy_kwargs in containers.py), while _status_from_exited_container treats restart_count < backoff_limit with backoff_limit=0 as exhausted → terminal FAILED + release_gpu. Backend can free the GPU while Docker keeps restarting the container. Field has no ge=1 constraint.

Not attachable as an inline diff comment (unchanged on this head relative to main after #925). Tracking here so the restart-race gap list is complete.

Agent patch snippet:

In DeploymentConfig `backoff_limit` (`entities.py` / schema): add
`Field(ge=1, ...)` (or document and map `0` explicitly). Align Docker
`MaximumRetryCount` semantics with status mapping in
`_status_from_exited_container` so unlimited-retry Docker behavior cannot
pair with immediate GPU release. Add a unit test for `backoff_limit=0`.
Optional follow-up; do not block the Envoy/keep-alive fix on this.

[tylersbray]

agent (): Posted verified findings R1–R7 as inline threads (R6 already as issue comment).