Repository navigation
fix(intercept): resolve bridge's own lookups without the tunnel it's redialing - #20
Merged
Merged
Conversation
…redialing When the gRPC connection to the proxy drops, the k8spf redial needs real DNS: resolvePod → K8s API → aws eks get-token → STS. The _bridge group was meant to skip the DNS redirect for this, but since /etc/resolv.conf names only 127.0.0.1, its lookups still reached the DNS server that asks the tunnel. With --forward-domains "*" that includes the API server itself, so every redial failed and the intercept never recovered: all DNS in the container stayed SERVFAIL until it was torn down. - Start a bypass DNS server that only asks the original nameserver, and redirect the _bridge group's UDP :53 to it instead of RETURN. The app's lookups still go through the tunnel, with no fallback around it. - Replace a pooled SPDY port-forward dialer once its connection closes. The pool handed the dead one back forever, so redials would fail even with DNS working. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'t apply GitHub runners give containers an Azure search domain, and busybox nslookup then queries only witness.<domain>.internal.cloudapp.net. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When an intercept's gRPC connection to the proxy drops, it never comes back, and every DNS lookup in the container fails until the bridge is torn down.
The redial goes
resolvePod → K8s API → aws eks get-token → STS, so it needs real DNS. The_bridgegroup'sRETURNrule (551b34d) was meant to give it that. Since #9, though,/etc/resolv.confnames only127.0.0.1. So bridge's own lookups still reach the DNS server that asks the tunnel, and with--forward-domains "*"that includes the API server. The redial can't resolve the cluster it's redialing, and it fails forever:This takes down vercel/api's Devbox E2E, which bridges api-devbox and subscriber-devbox into staging. In runs 42716374698 and 42136406829, both bridges lost their connection within 35s of each other. DNS then failed about 12,000 times with no recovery, and about 70 tests failed (
getaddrinfo EAI_AGAIN vercel.com, S3, DynamoDB, ElastiCache…).Change
_bridgegroup's UDP :53 is redirected to it instead ofRETURN. The app's lookups are unchanged: they still go through the tunnel, with no fallback around it.dialerPoolhanded a cachedportforward.Dialerback forever, even after its SPDY connection closed. So once DNS works, every redial to the same pod would still fail. It now replaces a dialer whoseCloseChan()has fired.Validation
TestBridgeResolvesWhileTunnelIsDownruns intercept undersg _bridge(as the feature does) and stops the proxy. Then it checks two things: bridge's group still resolves a name only the container's own Docker DNS knows, and root (the app) doesn't. Onmainit fails withCan't find witness: No answer, the same as CI. With this change it passes._bridgerule comes before the general redirect, with noRETURN) and for replacing a closed pooled dialer.🤖 Generated with Claude Code