Fix GCP Consul DNS resolution with two-layer protection #1430
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add dpkg-divert to permanently block gce-resolved.conf in Packer image - Configure Consul as recursive DNS with GCE forwarder in startup script - Remove routing domains approach (Domains=~consul) which doesn't work reliably - Restart systemd-resolved after Consul starts to prevent marking DNS as unreachable
Member
Provision script here is GPC specific already so we can merge it without IFs for different clouds. Later when merging configuration related on BYOC and multi-cloud support we can resolve it. |
tomassrnka
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
jakubno
requested changes
Nov 3, 2025
Replace sleep 3 with 10x 1s wait for consul to start
djeebus
reviewed
Nov 3, 2025
| # Give Consul a moment to start its DNS server on port 8600 | ||
| echo "- Waiting for Consul DNS to start on port 8600..." | ||
| for i in {1..10}; do | ||
| if nc -z 127.0.0.1 8600 2>/dev/null; then |
Contributor
There was a problem hiding this comment.
nslookup returns non-zero if the lookup fails, could use that for a more meaningful check
Suggested change
| if nc -z 127.0.0.1 8600 2>/dev/null; then | |
| if ! nslookup google.com; then |
djeebus
reviewed
Nov 3, 2025
| echo "- Restarting systemd-resolved to apply Consul DNS config" | ||
| systemctl restart systemd-resolved | ||
| echo "- Waiting for systemd-resolved to settle" | ||
| sleep 5 |
Contributor
There was a problem hiding this comment.
Could use nslookup here as well to wait for it to successfully resolve names
Replaced sleep with actual check on systemd-resolved restart
jakubno
approved these changes
Nov 4, 2025
tomassrnka
added a commit
that referenced
this pull request
Nov 5, 2025
Changes fixed ubuntu 22.04 image to latest ubuntu 22.04 lts image family, this is follow up on #1430 dns-fix.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: This works only for GCE deployment and does not take into account AWS / BYOC. The IP address is the same in AWS, so actually it should work, but it's just not generic enough at this moment
Note
On GCE, block gce-resolved.conf at image build and start Consul with a dynamically fetched GCE DNS recursor, reconfiguring systemd-resolved after Consul is up so Consul handles all DNS.
dpkg-divertto blockgce-resolved.conf(/etc/systemd/resolved.conf.d/gce-resolved.conf) to avoid DNS conflicts with Consul.start-client.sh):systemd-resolvedto use only127.0.0.1:8600; remove routing domains and disable GCE'sgce-resolved.conf.--recursor.systemd-resolved; add DNS readiness checks and cache flush.Written by Cursor Bugbot for commit 9fbd28e. This will update automatically on new commits. Configure here.