Skip to content

gnmi: keep stream sync alive across target resubscribes - #547

Open
steiler wants to merge 1 commit into
mainfrom
fix/gnmi-stream-resubscribe
Open

steiler wants to merge 1 commit into
mainfrom
fix/gnmi-stream-resubscribe

Conversation

@steiler

@steiler steiler commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Refs #542 (gNMI half; the NETCONF reconnect is handled separately, so this does not close the issue).

Problem

gnmiSubscribe returned on the first error reported by the target's subscription. gnmic resubscribes on its own and uses that channel only to report each failure, so after a device restart nothing consumed the subscription any more. Running was never refreshed and no deviation was detected, while the connection status still read connected. Restarting data-server was the only way out.

Change

  • Errors from the target's subscription are logged and no longer end the sync.
  • After a failure, the next initial sync is a full snapshot. It is committed with the configured sync paths as deletes plus the import (the same pattern the NETCONF sync uses), so config lost on the device surfaces as a deviation instead of being merged into stale state. An empty snapshot still clears the paths; previously it was swallowed as "sync no config changes".
  • The resubscribe and sync-response events travel through the notification channel behind all earlier notifications, after in-flight notification processing finishes. This also closes an existing race where SyncResponse could overtake notifications, which would now matter because a replace with a partial snapshot would drop data.
  • Ticker commits are held back until the snapshot arrives, and commits reach running in the order they were started.

Behaviour to be aware of

  • Sync paths with wildcard keys are skipped by the replace delete (logged at debug), as in the NETCONF sync, so stale entries under them can remain.
  • A sync path that does not parse as an sdcpb path is logged and left out of the replace; it is still subscribed. Start does not fail.
  • Every failed subscribe attempt while the device is down discards the accumulated partial tree.

Tests

  • TestGnmiSubscribe_StreamErrorKeepsSyncAlive
  • TestStreamSync_ResubscribeReplacesSyncedPaths (first sync merges only; resync with empty device config clears the paths; resync with config imports it)

Both fail without the change. go test ./pkg/datastore/target/gnmi/ -race passes; not verified against a real gNMI device.

gnmiSubscribe returned on the first error reported by the target's
subscription. The target resubscribes on its own and uses that channel
only to report each failure, so after a device restart nothing consumed
the subscription any more: running was never refreshed and no deviation
was detected, while the connection status still read connected.

Treat those errors as non-fatal. Following a failure, the next initial
sync is a full snapshot and replaces the configured sync paths in
running, so config lost on the device surfaces as a deviation instead of
being merged into stale state. An empty snapshot still clears the paths.

The resubscribe and sync-response events now travel through the
notification channel behind all earlier notifications, ticker commits
are held back until the snapshot arrives, and commits reach running in
the order they were started.

Signed-off-by: Markus Vahlenkamp <markus.vahlenkamp@nokia.com>
@steiler
steiler requested a review from a team as a code owner November 12, 2026 14:48
@codecov

codecov Bot commented Nov 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.00000% with 19 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/datastore/target/gnmi/stream.go 75.00% 11 Missing and 8 partials ⚠️

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant