Docs update of the proposed HA solution of node local dns in PVS mode of kube-proxy

[carterzhao]

The proposed HA solution of node local dns has been given here. But It will not work in IPVS mode of kube-proxy. I want to know whether there are some plans to support the HA solution in IPVS mode of kube-proxy.
Looking forward to your reply！！

[bowei]

Any ideas would be welcome :-)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[bowei]

/lifecycle-remove stale
/lifecycle frozen

[prameshj]

For IPVS mode, the HA solution is to run 2 replicas as described in the KEP:

Running 2 daemonsets of node-local-dns using the same listenIP - 169.254.20.10 via SO_REUSEPORT option. Upgrades will be done one daemonset at a time.

I see this got removed from the KEP when merging 2 KEPs together in kubernetes/enhancements#2487

[bowei]

Pavithra -- can you send a PR to add back to the KEP?

[bowei]

Also -- it probably needs to have documentation if this is a recommended setup...

[prameshj]

Pavithra -- can you send a PR to add back to the KEP?

Yes, already created - kubernetes/enhancements#2592

Also -- it probably needs to have documentation if this is a recommended setup...

I agree we need better documentation of this. It isn't necessarily the recommended setup, since it is twice the resources and requires managing conflicts - only one replica should handle interface/iptables management.
I will use this issue to track documentation.

[mlowery]

@prameshj: Regarding your comment:

requires managing conflicts - only one replica should handle interface/iptables management

Are there known issues where the two replicas will step on each other? If so, can you point me to them? When I look at the code, it seems idempotent and thread-safe. Is it not sufficient to simply pass --skipteardown to allow the replicas to safely coexist?

[prameshj]

@prameshj: Regarding your comment:

requires managing conflicts - only one replica should handle interface/iptables management

Are there known issues where the two replicas will step on each other? If so, can you point me to them? When I look at the code, it seems idempotent and thread-safe. Is it not sufficient to simply pass --skipteardown to allow the replicas to safely coexist?

I missed this comment.. apologies for the super late reply. You could pass --skipteardown to both replicas, but then the iptables rules and nodelocaldns interface would need to be torn down. This is applicable mostly for cases where nodelocaldns is being disabled. It is probably not a huge issue in IPVS mode(where only a link local IP is used) if cleanup is skipped, since no other service uses that same link local IP. So, even if iptables rules take it nowhere, nothing will break, if pods have already switched to using kube-dns service IP.

For upgrades, since only one replica upgrades at a time, --skipteardown on both replicas should be ok.

However, if kube-dns service VIP is reused for nodelocaldns(in order to cleanly fallback to kubedns when nodelocaldns is down/disabled), then skipping cleanup will blackhole DNS traffic.