fix(healthcheck): guard periodic checks against divide-by-zero when config is cleared#3400
fix(healthcheck): guard periodic checks against divide-by-zero when config is cleared#3400mvanhorn wants to merge 1 commit into
Conversation
| c.configMutex.Lock() | ||
| defer c.configMutex.Unlock() |
There was a problem hiding this comment.
Because above we run c.stop() to cancel the long running goroutine and wait for <-c.done, the long running goroutine should be finished and so there should be no need to hold a mutex here I think. I'll dig into this, to avoid these AI-bandaid-fixes (that do the job, but it does bloat code whereas it does not fix the real source of the problem)
There was a problem hiding this comment.
You're right that after c.stop() and <-c.done the periodic goroutine can't touch these fields, so the lock in Stop isn't about that goroutine. What it guards is SetConfig: Stop runs on the collectHealthErrors goroutine (started at tunnelup.go:107, calls Stop at 148/156) while SetConfig runs on the VPN loop goroutine in onTunnelUp (tunnelup.go:88). If the loop state machine already guarantees those never overlap across a restart cycle, then the Stop hunk is redundant and I'm happy to drop it.
The part I'd keep either way is fullPeriodicCheck: on master it indexes c.tlsDialAddrs unlocked, while smallPeriodicCheck copies under the same mutex for the same fields, so this just makes the two checks consistent. The empty-slice guards also protect the try%len modulo from a divide-by-zero panic when the config is empty.
Happy to trim the PR to whichever subset you want after you've dug in.
Description
The healthcheck goroutine can panic with
panic: runtime error: integer divide by zero(stack atinternal/healthcheck/checker.goinfullPeriodicCheck).fullPeriodicCheckindexed the shared config slice directly withc.tlsDialAddrs[try%len(c.tlsDialAddrs)].Checker.Stop()clears the config by settingc.tlsDialAddrs = nil(andicmpTargetIPs/smallCheckType) — but did so without holdingc.configMutex. AfullPeriodicCheckstill in flight during or just afterStop()then evaluatestry % len(c.tlsDialAddrs)withlen == 0and panics. It is also a data race:fullPeriodicCheckreadc.tlsDialAddrswithout the mutex whileStop/SetConfigwrote it.smallPeriodicCheckalready copiesc.icmpTargetIPsunder the mutex, but itsicmpTargetIPs[try%len(icmpTargetIPs)]had the same latent divide-by-zero when that copy is empty (StopnilsicmpTargetIPstoo).This change:
Stop()now takesc.configMutexwhile clearing the config, synchronizing the clear with the readers.fullPeriodicCheck()copiesc.tlsDialAddrsinto a local slice underc.configMutex(mirroringsmallPeriodicCheck) and returns early when the copy is empty — there is nothing to dial — instead of dividing by zero. The check closure indexes the local copy.smallPeriodicCheck()returns early when the (already-copied) ICMP target slice is empty and the check type is not DNS, so the copied-slice modulo cannot divide by zero either. The DNS-fallback path is unchanged.Behavior is identical for a normally-running checker with non-empty config; the guards only affect the teardown / empty-config window.
Verified with
go test ./internal/healthcheck/...,go test -race ./internal/healthcheck/...,go build ./..., andgofmt -l(clean). A regression test exercises the periodic checks with empty target slices and asserts they no longer panic.Issue
Closes #3398
Assertions