fix(publicip): retry fetching information on failure - #3388
Conversation
|
Thanks for the contribution. Unfortunately that's a lot of added complexity for a bandaid fix which does not address the true issue here.
It should not be "flaky". The "full" healthcheck (TLS+DNS check) passed before fetching the public ip information, so the tunnel should be already working well. If there was an outage or unhealthy connections beforehand, it shouldn't matter for this new connection. On top of this, in issue #3300 you can see there are other issues such as If this is still unresolved with the latest image which should have a 15s timeout (double check with log timestamps), we can proceed with the bandaid fix for the public ip but with a fixed number of retries (3) and no retry wait time, to keep code simple - if it still fails after 3 x 15s retries, it probably won't work ever. |
|
Thanks for your feedback! To be more precise, the few times I observed this behavior was when AirVPN had an outage. And I think the "flakiness" was on their side (due to everyone trying to reconnect to the serve rthat just went up at the same time). I will re-check my logs when I get back home, and report back. I remember I am using 3.41.1, which I believe is the last release. |
|
try the latest image It's about 7 months ahead of v3.41.1 and not the same as v3 or v3.41.1 |
|
So after one week of running the latest image (I was previously pinned to 3.41.1), I haven't had the issue anymore. I'll close this PR as unneeded, and comment back if I ever encounter the problem again. Thanks! |
|
Reopening: I got this again on the latest image (commit 78f6076, built 2026-07-19). This was the AirVPN outage scenario I mentioned earlier. It started here, with the small ICMP check failing, followed by path MTU discovery getting nothing back and a TCP dial to a bare IP timing out: Nothing was reachable through the tunnel for about ten minutes, by name or by IP, and the VPN restarted 30 times in that window. Then the connection came back: The tunnel passed its startup healthcheck at 01:56:30, and the public IP fetch still failed about 15 seconds later on all four APIs. The 15s timeout is clearly applied and does not help, because the connection was not slow, it was My guess is that this is a thundering herd effect: when an AirVPN server comes back, every client that was on it reconnects at the same time, and the exit stays congested for a short while afterwards. That would explain why three of Worth noting that the periodic healthcheck already does exactly what this PR proposes: withRetries with escalating 10s, 15s, 30s timeouts, with the comment "20s timeout in case the connection is under stress". The public IP fetch is a What do you think about that analysis? I can do other tests if you have other ideas. Also happy to rewrite this PR as you suggested, with 3 fixed retries and no wait time (althouhg in a thundering herd problem, adding some wait could be beneficial). Let me know and I will force-push. |
|
On addressing the root cause, here is a proposal that also covers the version check. Tell me if you like it before I write it. Both failing (cannot get version information from #3000, and mine) fetches sit in Proposal: promote your Three users: healthcheck unchanged with {10s, 15s, 30s} and no waits, publicip wrapping I looked at putting a retrying This also lets me drop the If the shape works I will rewrite the PR this way. If you would rather keep it to publicip with 3 fixed retries and no shared package, say so and I will do that instead. Or if you have other ideas, I'm also interested :) |
Description
The public IP information is fetched only once per tunnel-up (
onTunnelUp→RunOnce). That single fetch runs seconds after reconnection, when the tunnel is often still flaky, and if it fails, the data stays empty until the next tunnel-up, potentially days later, with/v1/publicip/ipserving{"public_ip":""}the whole time (breaking Homepage widgets and monitoring, as reported in #3300).Real-world example (v3.41.1, WireGuard/AirVPN): after a ~25 min provider outage, the healthcheck-triggered reconnect succeeded, but the tunnel-up fetch timed out on all four sources (
all fetchers failed: … Client.Timeout exceeded). The tunnel stabilized seconds later, yet the IP stayed empty for days because nothing ever retried.This PR makes the fetch self-healing:
ClearData) or if the public IP feature is disabled.No new settings (no wiki change needed), and zero extra API load when fetches succeed. Includes two unit tests (pass with
-race);golangci-lintclean;docker build .succeeds.Issue
Fixes #3300 (also previously reported as #1855, #2848).
Assertions