Failed to read message: nats: connection closed

[static-moonlight]

I implemented a simple pipeline to consume NATS messages and do something with them. This was done as a replacement of the NATS client to increase stability and reliability.

I was surprised when during my first test (shutdown the NATS server for a couple minutes and start it again), Redpanda Connect (RPC) will never automatically recover from that.

Even after the NATS server was online again for at least 10 minutes, RPC continues to print this error:

level=error msg="Failed to read message: nats: connection closed" @service=redpanda-connect label="" path=root.input

I don't see anything in the CLI help to tell RPC how to handle such errors (like trying to reconnect, or just quit the process, or something). I also can't see anything in the documentation for nats_jetstream about error handling or auto reconnect.

I was under the impression that auto-reconnect in NATS is pretty much standard:

All the client libraries maintained on the nats.io GitHub page will automatically attempt to re-connect if their current server connection gets disconnected for any reason.

This is the important part of my pipeline config:

input:
  nats_jetstream:
    urls: [nats://${NATS_USERNAME}:${NATS_PASSWORD}@${NATS_ENDPOINT}]
    subject: ${NATS_SUBJECT}
    stream: ${NATS_STREAM}
    deliver: ${NATS_DELIVER}
    durable: ${NATS_CONSUMER}

[...]

I could need some advice about how to increase resilience for this one.

I need one of this:

1. RPC automatically reconnect after the NATS server recovered (no matter if this happens after 1 second/minute/hour/day/...)
2. RPC quit after a certain input error threshold was reached
3. Some form of command that can be used in Kubernetes (liveness probe) and Podman (health command) to check for errors and if necessary forcefully shutdown the container (running RPC)

2 and 3 would take advantage of the container restart policy to implement auto-recovery.

[mihaitodor]

Hey @static-moonlight, thank you for reporting this! I think that might happen because we're not checking https://pkg.go.dev/github.com/nats-io/nats.go#Subscription.IsValid here. I'll push a change with this fix when I get a chance, hopefully before the next release.