fix(stargate): back off discovery and report registration failures

[barrygreengus]

Why

Persistent Kubernetes API errors were retried immediately. Pylon registration and discovery logged certificate failures but discarded other connection, authentication, configuration, and stream errors, obscuring recovery failures.

Examples

• Persistent discovery failure: when the Kubernetes API returns 403 repeatedly, polling the watcher again without backoff immediately issues another failing request. A persistent permission error can therefore create a request and log loop. Applying kube's watcher backoff delays subsequent attempts; a fake API and paused clock verify that a second request is not issued immediately.
• Registration diagnosis: a refused connection or rejected authentication previously followed the reconnect path without reporting the non-certificate cause. Operators could see that registration was absent without knowing which operation failed. Warnings now identify the failed operation, a safe endpoint value, and error cause. Each failed retry emits a warning so a persistent problem remains visible. Existing retry delays bound the retry frequency. Certificate-specific ERROR diagnostics retain their existing duplicate suppression and corrective guidance.
• Closed streams: a registration stream that ends unexpectedly is now reported and followed by a reconnect delay, rather than silently entering another immediate registration cycle.
• Secret-file errors: malformed registration-token JSON can make a parser error include an excerpt of the file. The new diagnostic path omits that excerpt so reporting the failure does not expose token-file contents. Certificate-specific diagnostics and their existing guidance are preserved.

A missing token file now retains the operating system's not-found cause instead of logging only "failed to read". HTTP token-issuer failures retain their transport causes without printing credentials or sensitive query strings from the issuer URL. Redaction happens when logging, so callers retain the original error chain.

What changed

• Apply kube's default watcher backoff.
• Emit warnings directly at discovery and registration failure sites with tracing macros. Remove the logging dispatcher and the custom connection-log macro. Shared formatting only returns safe endpoint and error values; it does not log or manage retry state.
• Report unexpected stream completion directly, without constructing an artificial I/O error, and delay reconnects after registration-stream loss.
• Preserve token-resolution error chains and redact secret-file parser excerpts and sensitive HTTP URLs when rendering diagnostics.

Customer Release Notes

Stargate discovery backs off during Kubernetes API failures. Pylon reports actionable registration failures on every failed attempt without exposing token-file contents.

Plan Summary

Not applicable.

Usage

Registration warnings include the failed operation, a sanitized endpoint, the cause, and the cluster ID when available. Certificate-specific ERROR diagnostics retain their existing categories, corrective guidance, and duplicate suppression. Retry warnings remain visible during persistent failures.

Testing

All 515 pylon-lib unit tests and both public API tests passed with Cargo in a fresh isolated worktree. Clippy passed for all pylon-lib targets with warnings denied. The retry regression runs the registration loop against a refused connection and verifies repeated warnings, the cause, cluster context, and endpoint redaction. Tests also cover token-file excerpts, sensitive HTTP URLs, gRPC metadata and binary details, and existing certificate diagnostics. Formatting, whitespace, and skill-fanout checks passed.

Full workspace and Bazel suites were not rerun for this internal logging change; no build declarations or dependencies changed. No live cluster was used.

Notes

Layer 3 of the Stargate maintenance stack. Repeated warnings are intentional and follow the existing retry cadence. Connection DEBUG events and new WARN events use a sanitized endpoint field; the existing certificate ERROR fields remain unchanged. No new metrics or configuration controls.

Issues

Relates to #1817

References

Kube watcher backoff

Related Pull Requests

Depends on #1820. Next layer: #1826.

Dependencies

No added or upgraded dependency versions. Enabled the existing Tokio test-util feature for deterministic watcher tests and refreshed Bazel metadata. No license or NOTICE changes.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🛡️ CodeQL Analysis

🚨 Found 5 issue(s)

Severity Breakdown:

• 🔴 Errors: 0
• 🟡 Warnings: 0
• 🔵 Notes: 0

📋 Top Issues

🔗 View full details in Security tab

_{🕐 Last updated: 2026-09-11 22:27:59 UTC | Commit: b4c0176}