Add kindnet network plugin #17158

aojea · 2024-12-29T18:11:56Z

Kindnet has been running in Kubernetes CI for a while and there are some people that uses it, I've been adding new features like dns cache or kernel bypass or admin network policies that are not present in all the other common cnis.

Integration with kops will help to improve its testing and also benefits users that are looking for a more minimalistic CNI plugin

https://github.com/aojea/kindnet

aojea · 2024-12-29T18:12:25Z

/assign @justinsb @hakman

pkg/apis/kops/networking.go

cmd/kops/integration_test.go

aojea · 2024-12-30T18:03:28Z

Ok, it is working now

 kops validate cluster --wait 10m
Using cluster from kubectl context: myclustername.kindnet.io

Validating cluster myclustername.kindnet.io

I1230 18:02:56.188347 3334459 gce_cloud.go:307] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b]
INSTANCE GROUPS
NAME                            ROLE            MACHINETYPE     MIN     MAX     SUBNETS
control-plane-us-central1-c     ControlPlane    e2-medium       1       1       us-central1
nodes-us-central1-c             Node            e2-medium       1       1       us-central1

NODE STATUS
NAME                                    ROLE            READY
control-plane-us-central1-c-bw8j        control-plane   True
nodes-us-central1-c-3rkx                node            True

Your cluster myclustername.kindnet.io is ready

aojea · 2024-12-31T13:32:30Z

failed job is unrelated

Kubernetes e2e suite: [It] [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support readOnly directory specified in the volumeMount expand_less

This is ready for review

hakman

LGTM, just a few comments. Thanks @aojea!
I will send a PR for a pre-submit for kindnet.

cmd/kops/integration_test.go

pkg/apis/kops/networking.go

rifelpet

We haven't used it in a while but we may want to put this behind a kops feature gate to make it clear that it is experimental and we may make breaking changes. WDYT @hakman ?

pkg/model/components/kindnet.go

aojea · 2024-12-31T15:21:02Z

We haven't used it in a while but we may want to put this behind a kops feature gate to make it clear that it is experimental and we may make breaking changes. WDYT @hakman ?

kindnet is already used in CI for other kubernetes jobs and projects https://grep.app/search?q=aojea/kindnet so breaking compatibility within kindnet is something I don't expect, however, how to get the better integration with Kops is something I need your advice

hakman · 2024-12-31T15:21:19Z

We haven't used it in a while but we may want to put this behind a kops feature gate to make it clear that it is experimental and we may make breaking changes. WDYT @hakman?

I think we cab skip the flag, as long as we add a warning in the doc file.
Also, may be a good idea to add a mention here:
https://kops.sigs.k8s.io/networking/#supported-networking-options

hakman · 2025-01-06T13:24:27Z

@aojea Not sure if Kindnet need the CNI Network Plugin binaries.
If it doesn't require those binaries in /opt/cni/bin/ please add Kindness to this function:

kops/pkg/apis/kops/cluster.go

Lines 941 to 945 in 84948bb

    
           func (c *Cluster) InstallCNIAssets() bool { 
        
           	return c.Spec.Networking.AmazonVPC == nil && 
        
           		c.Spec.Networking.Calico == nil && 
        
           		c.Spec.Networking.Cilium == nil 
        
           }

Besides the few remaining nits and the 3 failing tests, I think this is pretty good as it is.
The 3 tests can also be fixed at a later time, as Kindnet is marked as experimental.

aojea · 2025-01-07T09:44:18Z

@aojea Not sure if Kindnet need the CNI Network Plugin binaries. If it doesn't require those binaries in /opt/cni/bin/ please add Kindness to this function:

kops/pkg/apis/kops/cluster.go

Lines 941 to 945 in 84948bb

func (c *Cluster) InstallCNIAssets() bool {

return c.Spec.Networking.AmazonVPC == nil &&

c.Spec.Networking.Calico == nil &&

c.Spec.Networking.Cilium == nil

}

Besides the few remaining nits and the 3 failing tests, I think this is pretty good as it is. The 3 tests can also be fixed at a later time, as Kindnet is marked as experimental.

great, I will address last comments and work on this today

/test pull-kops-e2e-cni-kindnet

aojea · 2025-01-07T15:50:42Z

/test pull-kops-e2e-cni-kindnet

get more networking information usefult to troubleshoot network issues.

add kindnet as an experimental network addon containerd adds the requirement to use the loopback cni plugin, kindnet provides that capability and containerd does not require it since containerd/containerd/pull/10238 Change-Id: I1397a90186885b02e98b5ffa444fe629c1046757

aojea · 2025-01-08T01:09:59Z

@aojea Not sure if Kindnet need the CNI Network Plugin binaries.
If it doesn't require those binaries in /opt/cni/bin/ please add Kindness to this function:

it needs the loopback plugin so it does not hurt having those

Besides the few remaining nits and the 3 failing tests, I think this is pretty good as it is. The 3 tests can also be fixed at a later time, as Kindnet is marked as experimental.

I've added some rudementary network connection logging to kindnet that is very useful ,

Checking the test failed t] [sig-network] Networking Granular Checks: Services should be able to handle large requests: http , it tries to connect to &protocol=http&host=100.70.71.8&port=80&tries=1

The connections are done from

I0107 18:58:52.119749 49720 resource.go:175] test-container-pod i-09094242e1b8c58f1 Running [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2025-01-07 18:52:27 +0000 UTC } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2025-01-07 18:52:26 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2025-01-07 18:52:27 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2025-01-07 18:52:27 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2025-01-07 18:52:26 +0000 UTC }]

node i-09094242e1b8c58f1 , the kindnet pod on that node (it is on the test output log) is https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kops/17158/pull-kops-e2e-cni-kindnet/1876696173184552960/artifacts/cluster-info/kube-system/kindnet-kndvd/kindnet-cni.log

we can see the connections go through

I0107 18:54:02.672517 1 conntrack.go:175] Connection finished: [EventDestroy] Timeout: 0, <tcp, Src: 100.96.4.80:43422, Dst: 100.70.71.8:80>, Zone 0, Acct: [orig: 3 pkts/152 B] [reply: 2 pkts/112 B], , , , ,

but practically no bytes, the connection is established but closed immediately

I will work on those tests after this merge, it seems specific of this environment as I could not repro locally and in other environments, so @hakman this should be ready to go

hakman · 2025-01-08T04:51:21Z

Thanks @aojea! 🙂
/lgtm
/retest
/assign @justinsb

hakman · 2025-01-08T05:42:22Z

/retest

aojea · 2025-01-08T13:03:12Z

/test pull-kops-e2e-cni-kindnet

hakman · 2025-01-08T13:52:08Z

/test pull-kops-e2e-cni-kindnet

All green now!!! 👏

aojea · 2025-01-08T13:56:10Z

/test pull-kops-e2e-cni-kindnet

All green now!!! 👏

it was an mtu issue :) aojea/kindnet#143

hakman · 2025-01-08T13:57:35Z

/test pull-kops-e2e-cni-kindnet

All green now!!! 👏

it was an mtu issue :) aojea/kindnet#143

but could have been DNS 🤣

justinsb · 2025-01-08T14:03:05Z

docs/networking/kindnet.md

+  --zones $ZONES \
+  --networking kindnet \
+  --yes \
+  --name myclustername.mydns.io


Nit: we normally use example.com, I think that is reserved for examples

justinsb · 2025-01-08T14:04:49Z

k8s/crds/kops.k8s.io_clusters.yaml

@@ -5773,6 +5773,39 @@ spec:
                    description: GCPNetworkingSpec is the specification of GCP's native
                      networking mode, using IP aliases.
                    type: object
+                  kindnet:
+                    description: KindnetNetworkingSpec configures Kindnet settings.


Nit: an annoying thing about go docs mapping to OpenAPI docs is that we probably don't want to follow normal go conventions for comments. The reader of the OpenAPI docs doesn't see the name KindnetNetworkingSpec on the struct, because OpenAPI doesn't have structs (or we don't use them)

justinsb · 2025-01-08T14:08:43Z

pkg/apis/kops/validation/validation.go

+	if v.Kindnet != nil {
+		if optionTaken {
+			allErrs = append(allErrs, field.Forbidden(fldPath.Child("kindnet"), "only one networking option permitted"))
+		}


We probably should be setting optionTaken here, but we're also missing it on LyftVPC and GCP. I'll send a fix separately :-)

justinsb · 2025-01-08T14:10:54Z

pkg/apis/kops/validation/validation.go

+func validateNetworkingKindnet(cluster *kops.Cluster, v *kops.KindnetNetworkingSpec, fldPath *field.Path) field.ErrorList {
+	allErrs := field.ErrorList{}
+
+	if v.Masquerade != nil && v.Masquerade.Enabled != nil && *v.Masquerade.Enabled {


Nit: We have ValueOf, v.Masquerade.Enabled != nil && *v.Masquerade.Enabled === fi.ValueOf(v.MasqueradeEnabled)

justinsb · 2025-01-08T14:11:35Z

pkg/dump/dumper.go

@@ -345,7 +345,19 @@ func (n *logDumperNode) dump(ctx context.Context) []error {
 	if err := n.shellToFile(ctx, "sudo iptables -t filter --list-rules", filepath.Join(n.dir, "iptables-filter.log")); err != nil {
 		errors = append(errors, err)
 	}
-	if err := n.shellToFile(ctx, "ip route", filepath.Join(n.dir, "ip-routes.log")); err != nil {
+	if err := n.shellToFile(ctx, "sudo nft list ruleset", filepath.Join(n.dir, "nftables-ruleset.log")); err != nil {


Awesome 👍

justinsb · 2025-01-08T14:12:30Z

pkg/model/components/kindnet.go

+			c.Masquerade.NonMasqueradeCIDRs = append(c.Masquerade.NonMasqueradeCIDRs, clusterSpec.Networking.PodCIDR)
+		}
+		if clusterSpec.Networking.ServiceClusterIPRange != "" {
+			c.Masquerade.NonMasqueradeCIDRs = append(c.Masquerade.NonMasqueradeCIDRs, clusterSpec.Networking.ServiceClusterIPRange)


I like making this explicit 👍

justinsb · 2025-01-08T14:20:17Z

pkg/model/iam/iam_builder.go

@@ -777,6 +789,12 @@ func addKubeRouterSrcDstCheckPermissions(p *Policy) {
 	)
 }

+func addKindnetSrcDstCheckPermissions(p *Policy) {
+	p.unconditionalAction.Insert(
+		"ec2:ModifyInstanceAttribute",


Not a blocker, but this has security/performance implications IIRC. Do we need this permission in all modes? (e.g. if we're running IPv6, do we need it?)

justinsb · 2025-01-08T14:24:55Z

This LGTM - awesome work @aojea 🎉 Only question is about those src/dest checks, if we can avoid needing to turn them off that would be awesome. But my 2c is that the easiest way to do this is to merge this, and then send a follow on PR to try them off and seeing what breaks, so I'm in favor of merging!

Any objections to merging @hakman / @rifelpet ?

hakman · 2025-01-08T14:28:20Z

Any objections to merging @hakman / @rifelpet ?

@justinsb 👍 , though, would like to add an IPv6 pre-submit if we want to not disable src/dest checks for that case. In any case, I think enabling IPv6 support will require some additional work.

aojea · 2025-01-08T14:33:35Z

In any case, I think enabling IPv6 support will require some additional work.

for kindnet it should work out of the box.
Cloud providers may need some changes, those were my tests of ipv6 with kubeadm in aws https://github.com/aojea/k8s-aws-ipv6 and in gce https://gist.github.com/aojea/b5c18f99ed048ef6a0c06640e3ab4de7

justinsb · 2025-01-08T14:33:41Z

would like to add an IPv6 pre-submit if we want to not disable src/dest checks for that case. In any case, I think enabling IPv6 support will require some additional work.

I think that's a great idea, and I think we should do that in follow-on PRs.

justinsb · 2025-01-08T15:24:13Z

OK I think we can get this in!

/approve
/lgtm

k8s-ci-robot · 2025-01-08T15:24:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 29, 2024

k8s-ci-robot requested review from johngmyers and zetaab December 29, 2024 18:12

k8s-ci-robot added area/addons cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. area/api area/documentation size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 29, 2024

k8s-ci-robot assigned hakman and justinsb Dec 29, 2024

aojea force-pushed the kindnet branch 2 times, most recently from 13985ad to 9e14fd2 Compare December 30, 2024 12:11

k8s-ci-robot added the area/nodeup label Dec 30, 2024

aojea force-pushed the kindnet branch from 9e14fd2 to 013059d Compare December 30, 2024 13:34

aojea commented Dec 30, 2024

View reviewed changes

pkg/apis/kops/networking.go Show resolved Hide resolved

aojea force-pushed the kindnet branch from 013059d to 5d0a2bd Compare December 30, 2024 13:50

aojea commented Dec 30, 2024

View reviewed changes

cmd/kops/integration_test.go Show resolved Hide resolved

aojea force-pushed the kindnet branch from 5d0a2bd to 6c80f14 Compare December 30, 2024 17:32

aojea force-pushed the kindnet branch from 6c80f14 to 44e8540 Compare December 31, 2024 12:08

aojea changed the title ~~[WIP] add kindnet network plugin~~ add kindnet network plugin Dec 31, 2024

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 31, 2024

hakman changed the title ~~add kindnet network plugin~~ Add kindnet network plugin Dec 31, 2024

hakman requested changes Dec 31, 2024

View reviewed changes

cmd/kops/integration_test.go Outdated Show resolved Hide resolved

pkg/apis/kops/networking.go Show resolved Hide resolved

aojea force-pushed the kindnet branch from 44e8540 to b6c2d3b Compare December 31, 2024 15:07

rifelpet reviewed Dec 31, 2024

View reviewed changes

pkg/model/components/kindnet.go Outdated Show resolved Hide resolved

aojea force-pushed the kindnet branch from 6ff40f3 to bf1b56d Compare January 7, 2025 09:34

aojea force-pushed the kindnet branch 2 times, most recently from 2bdbd59 to a36c140 Compare January 7, 2025 15:18

improve log dumper

5b08cef

get more networking information usefult to troubleshoot network issues.

aojea force-pushed the kindnet branch 3 times, most recently from 3827c06 to 0aae903 Compare January 8, 2025 00:58

aojea force-pushed the kindnet branch from 0aae903 to f2c239d Compare January 8, 2025 01:09

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 8, 2025

justinsb reviewed Jan 8, 2025

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 8, 2025

k8s-ci-robot merged commit 2db9dbc into kubernetes:master Jan 8, 2025
33 checks passed

k8s-ci-robot added this to the v1.32 milestone Jan 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kindnet network plugin #17158

Add kindnet network plugin #17158

aojea commented Dec 29, 2024 •

edited

Loading

aojea commented Dec 29, 2024

aojea commented Dec 30, 2024

aojea commented Dec 31, 2024

hakman left a comment

rifelpet left a comment

aojea commented Dec 31, 2024

hakman commented Dec 31, 2024

hakman commented Jan 6, 2025

aojea commented Jan 7, 2025

aojea commented Jan 7, 2025

aojea commented Jan 8, 2025 •

edited

Loading

hakman commented Jan 8, 2025

hakman commented Jan 8, 2025

aojea commented Jan 8, 2025

hakman commented Jan 8, 2025

aojea commented Jan 8, 2025

hakman commented Jan 8, 2025

justinsb Jan 8, 2025

justinsb Jan 8, 2025

justinsb Jan 8, 2025

justinsb Jan 8, 2025

justinsb Jan 8, 2025

justinsb Jan 8, 2025

justinsb Jan 8, 2025

justinsb commented Jan 8, 2025

hakman commented Jan 8, 2025

aojea commented Jan 8, 2025

justinsb commented Jan 8, 2025

justinsb commented Jan 8, 2025

k8s-ci-robot commented Jan 8, 2025

Add kindnet network plugin #17158

Add kindnet network plugin #17158

Conversation

aojea commented Dec 29, 2024 • edited Loading

aojea commented Dec 29, 2024

aojea commented Dec 30, 2024

aojea commented Dec 31, 2024

hakman left a comment

Choose a reason for hiding this comment

rifelpet left a comment

Choose a reason for hiding this comment

aojea commented Dec 31, 2024

hakman commented Dec 31, 2024

hakman commented Jan 6, 2025

aojea commented Jan 7, 2025

aojea commented Jan 7, 2025

aojea commented Jan 8, 2025 • edited Loading

hakman commented Jan 8, 2025

hakman commented Jan 8, 2025

aojea commented Jan 8, 2025

hakman commented Jan 8, 2025

aojea commented Jan 8, 2025

hakman commented Jan 8, 2025

justinsb Jan 8, 2025

Choose a reason for hiding this comment

justinsb Jan 8, 2025

Choose a reason for hiding this comment

justinsb Jan 8, 2025

Choose a reason for hiding this comment

justinsb Jan 8, 2025

Choose a reason for hiding this comment

justinsb Jan 8, 2025

Choose a reason for hiding this comment

justinsb Jan 8, 2025

Choose a reason for hiding this comment

justinsb Jan 8, 2025

Choose a reason for hiding this comment

justinsb commented Jan 8, 2025

hakman commented Jan 8, 2025

aojea commented Jan 8, 2025

justinsb commented Jan 8, 2025

justinsb commented Jan 8, 2025

k8s-ci-robot commented Jan 8, 2025

aojea commented Dec 29, 2024 •

edited

Loading

aojea commented Jan 8, 2025 •

edited

Loading