Kubernetes Egress

[sarnowski]

The goal is to know where network connections are going to outside of the cluster and authorizing those in advance. All connections that leave the cluster network need to be whitelisted. For that, I propose an Egress resource to specify the whitelist.

Requirements

• No network traffic going out of the cluster by default.
• Explicit whitelisting possible for the user.
• Explicit whitelisting possible for system bootstrapping (like Docker registry and OAuth).

Not a requirement:

• Fine granular rules per pod. Only on a cluster level.

Example specification

apiVersion: "zalando.org/v1"
kind: EgressRuleSet
metadata:
   name: my-app-targets
spec:
   targets:
   - mydependency1.example.com:443
   - mydependency2.example.com:443
   - *.example.org:80

In some cases, pinning it down to predetermined domains doesn't work. Examples would be crawler or applications that need to react on user input like webhooks. In this case, one needs a switch to allow everything:

apiVersion: "zalando.org/v1"
kind: EgressRuleSet
metadata:
   name: world-access
spec:
   targets:
   - *:80

Multiple rule sets can exist at the same time and a union of the targets would determine the set of the whole cluster.

Wildcarding everything is okay as its still an explicit choice that could be checked during deployment. Wildcarding ports should be discouraged (and even not implemented) unless we find any valid use case where this is not purely security insensitive.

Example integration

Since this would probably be implemented as an HTTP proxy, the integration pattern should be that the standard environment variable http_proxy is set by default in every container that starts without the user having to specify it.

Example implementation

One should set up a HA/scalable HTTP proxy like squid. In addition, an egress-controller should observe the EgressRuleSet resources and reconfigure the squid accordingly.

The AWS Security Group of all Kubernetes nodes would not have the default "allow outbound" rule so that every traffic going out would be dropped. The HTTP proxy would need to run outside of the security group in some kind of "DMZ" setup (like the ALBs and ELBs) where Kubernetes nodes can go to. The HTTP proxy server then itself has full outbound rules in its Security Group.

[sarnowski]

Another minor advantage that comes out of having an HTTP proxy in place would be the possibility to enable caching for certain resources. This would be a huge advantage for our future build systems that typically download massive amounts of static resources from various repositories (maven, apt, npm, ...). Having a caching proxy here would decrease build times immensely.

[sarnowski]

Another implementation hint:
https://aws.amazon.com/de/blogs/security/how-to-add-dns-filtering-to-your-nat-instance-with-squid/

[hwinkel]

Hi, How is going with that. We have currently a use case where we need to proxy out the egress traffic to give the traffic a defined (set) of IP(s). The Destination has some IP based Filter to only allow requests from this given IPs. To make it pretty we could envision to announce the Source IP of the Node running the egress proxy by a L3 Networking model. ?!?

[hjacobs]

@hwinkel we haven't started working on egress yet, but for your use case it could be a simple routing rule in AWS to go via NAT gateways with fixed IP.

[dolftax]

@hjacobs I was looking for something silimar. Point me to relevant meetings or docs? I can help you guys build this :)

[hjacobs]

FYI: we are currently experimenting with dante as a SOCKS server on AWS. It really looks promising and Java has automatic SOCKS support (https://docs.oracle.com/javase/7/docs/technotes/guides/net/proxies.html).

[shruti-palshikar]

Is the selective egress a supported functionality now?

[mikkeloscar]

@shruti-palshikar No, we have not done any work on this, and it's also not something we are planning right now since we don't need it ourselves at the moment :)

[szuecs]

@shruti-palshikar dependent on what you are searching for you might use calico or cillium and network policies, I know that there is some effort in sig-network specifying egress network policies.
@hwinkel shameless plug https://github.com/szuecs/kube-static-egress-controller , is a controller that will create a CF stack to create NAT GWs and create routing table entries to select only a list of target networks to route through the NAT GW.

[shruti-palshikar]

@mikkeloscar Thanks for the response. @szuecs : My usecase is to allow egress from pods to certain selected external domains. With the default policy of a namespace being deny-all egress traffic, I am looking for ways to whitelist a few domains that the pods are allowed to reach out to.

[szuecs]

@shruti-palshikar domains are not an internet routeable entity for egress nor ingress. It's based on Layer3 of the OSI model and DNS is used to resolve a name to an IP. IP is the routeable entity, which you might can do in kubernetes with the mentioned CNI plugins + egress-traffic-policy object, but this repository is not the right audience.