feat(node): isolate cocoon VMs at L2 via CNI portIsolation + macspoofchk#6
Open
tonicmuroq wants to merge 1 commit into
Open
feat(node): isolate cocoon VMs at L2 via CNI portIsolation + macspoofchk#6tonicmuroq wants to merge 1 commit into
tonicmuroq wants to merge 1 commit into
Conversation
Same-node VM-to-VM isolation moves from iptables + bridge-nf-call-iptables
to the CNI bridge plugin, baked into the generated conflist:
- portIsolation: kernel BR_ISOLATED per veth (blocks same-bridge VM-to-VM
unicast/ARP/broadcast at L2, gateway still reachable)
- macspoofchk: nftables source-MAC pin (anti MAC spoof / FDB hijack)
setupIPTables no longer enables bridge-nf-call-iptables: the FORWARD DROP
rules (--drop-cidr, e.g. the fleet VM supernet 172.22.0.0/16) target only
cross-node / off-bridge destinations, which are L3-routed and hit FORWARD
without br_netfilter. Drops ensureBridgeNFCall/readSysctl (now unused).
This avoids the br_netfilter footguns (conntrack table pressure incl. dropped
flows, host-global toggle affecting docker0). Same-node = L2 (portIsolation),
cross-node = L3 (FORWARD DROP), neither needs conntrack.
Out of scope (node provisioner): VM egress default-drop to GKE master /
kubelet:10250 / internal 10.0.0.0/8.
tonicmuroq
force-pushed
the
feat/cocoon-vm-l2-isolation
branch
from
a6f5f5a to
a723332
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Persist the cocoon VM↔VM isolation that's currently applied by hand on the internal-cocoon nodes into cocoon-net, using the CNI
bridgeplugin's L2 features + an L3 FORWARD drop — nobr_netfilter.writeCNIConflist(node/node.go): the generated30-cocoon-dhcp.conflistbridge plugin now setsportIsolation: true— kernelBR_ISOLATEDon every VM veth → same-node VM↔VM blocked at L2 (unicast + ARP + broadcast), gateway/DHCP/egress still reachable.macspoofchk: true— nftables (bridge family) source-MAC pin → anti MAC-spoof / FDB hijack. Stateless, no conntrack.setupIPTables(node/iptables_linux.go): no longer callsensureBridgeNFCall. TheFORWARD -i cni0 -d <CIDR> DROPrules (from--drop-cidr, e.g. the fleet VM supernet172.22.0.0/16) only match cross-node / off-bridge destinations, which are L3-routed and traverseFORWARDwithoutbr_netfilter. Removes now-unusedensureBridgeNFCall/readSysctl.Why not br_netfilter
Enabling
bridge-nf-call-iptablesto catch same-node L2 traffic has real footguns at scale: conntrack tracks flows before the FORWARD DROP (so even blocked VM↔VM flows consume conntrack slots), the defaultnf_conntrack_maxis undersized for hundreds of VMs/node, and it's a host-global toggle that also dragsdocker0through netfilter. Doing same-node isolation at L2 (portIsolation) sidesteps all of it; cross-node is L3 and never needed br_netfilter.Validation
Applied by hand and validated on internal-cocoon-node-1..6 (real Windows VMs, incl. hibernate/wake MAC-swap): same-node VM↔VM blocked (ARP too), cross-node blocked, VM→gateway/ingress(10.16.0.11)/internet/DNS OK, macspoofchk pins the guest's actual MAC and re-pins across wake, control→VM (guac RDP) + return traffic unaffected, 40/40 VMs Ready.
Out of scope
VM egress default-drop to the GKE master / kubelet:10250 / internal
10.0.0.0/8(requirement ③) is intentionally not here — that's node-level egress policy for the node provisioner (needs anESTABLISHED,RELATEDaccept before the 10/8 drop so control-plane→VM return traffic survives).Deploy note
Rules are live on the fleet by hand today; merging + rolling this makes them survive node / cocoon-net restarts (previously the daemon regenerated the conflist without these fields on every start).