Implementing CIS Benchmarks for Iptables

[ozmodiar192]

Hello,
we are starting to implement the CIS benchmarks for Bottlerocket. The CIS Benchmark document shows an example shell script with iptables commands like this:

# Ensure default deny firewall policy
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

Then it says:

The above script can be deployed through a bootstrap container, which is the
mechanism that Bottlerocket provides for running custom code during the boot process.

It is my understanding that executing IPTables commands in a bootstrap container to modify rules would not work. Is the CIS Benchmark misleading, or am I misunderstanding the nature of bootstrap containers? My understanding is that the Bottlerocket way of handing this would be to use the bootstrap container to write out rules to the appropriate file system location like /etc/sysconfig/iptables using the root file system mount.

[arnaldo2792]

Hi @ozmodiar192, after this change was merged bootstrap containers gained the capabilities required to modify iptables rules without requiring modifications in the root filesystem. Keep in mind that you will want to configure your bootstrap container as mode = always in order to apply the iptables changes on subsequent reboots.

[nike21oct]

no no sysctl is used for changing the kernel settings but this one [settings.bootstrap-containers.cis-bootstrap] is having source which is having bootstrap container image as described in the document https://catalog.us-east-1.prod.workshops.aws/workshops/165b0729-2791-4452-8920-53b734419050/en-US/10-regulatory-compliance/cis-bottlerocket-eks/build-bootstrap-image. you shared , I build the image and stored in source , it runs via terraform but did not change the iptables rule

[stmcginnis]

Can you share your Dockerfile and script for this bootstrap container?

Make sure whatever "registry-path/bottlerocket-benchmarking/bootstrap" actually points to is accessible.

You will likely want mode = "always" so the iptables settings are applied every time the node boots.

[nike21oct]

hi @stmcginnis , i used the docker file provided in this URL for creating bootstrap container "https://github.com/aws-samples/containers-blog-maelstrom/tree/main/cis-bottlerocket-benchmark-eks/bottlerocket-cis-bootstrap-image" and store the image in my private registry to gitlab and used below setting in user_data.

user_data = base64encode(<<EOT`
[settings.bootstrap-containers.cis-bootstrap]
source = "registry.devops.telekom.de/mavie/system/tools/bottlerocket-benchmarking/bootstrap"
mode = "always"
EOT
)

can you please guide what could be the issue ?

[stmcginnis]

Nothing obvious is standing out to me. Maybe try connecting (aws ssm start-session --target $INSTANCE_ID or SSH if set up), enter-admin-container to connect, the sudo sheltie to get a prompt as if you are on the OS itself. From there, you can try journalctl | grep boot and see if there are any error messages from the initialization that would point to what the failure might be.

If it is successful, you should see something like:

Aug 07 16:47:50 ip-192-168-42-247.us-east-2.compute.internal bootstrap-containers[1277]: 16:47:50 [INFO] bootstrap-containers started
Aug 07 16:47:50 ip-192-168-42-247.us-east-2.compute.internal bootstrap-containers[1277]: 16:47:50 [INFO] Mode for 'bootstrap' is 'always'
Aug 07 16:47:50 ip-192-168-42-247.us-east-2.compute.internal systemd[1]: Finished bootstrap container bootstrap.

For my case, here are all the steps I did to double check functionality.

I created a container using this Dockerfile:

FROM alpine
COPY ./bootstrap-script.sh /
RUN chmod +x /bootstrap-script.sh
RUN apk update && apk add bash && apk add iptables && apk add ip6tables
ENTRYPOINT ["/bootstrap-script.sh"]

That pulls in a simple script that I just made one small iptables change to see it working:

#!/usr/bin/env bash

# Flush iptables rules
iptables -F

iptables -I INPUT -p tcp -m tcp --dport 22222 -j ACCEPT

I built that container image and pushed it to my private registry.

One thing to note: I originally did have an issue here, so maybe double check if this might apply to you. My development machine is an ARM64 box, but my cluster nodes are AMD64. That caused an "exec format error" when it tried to run it.

I then launched instances with the following user data:

[settings.bootstrap-containers.bootstrap]
source = "id.dkr.ecr.us-east-2.amazonaws.com/bootstrap:latest"
mode = always

Made sure there were no errors in the system log, and double checked the iptables configuration was applied:

bash-5.1# iptables -L | grep 22             
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:22222

The only thing I see different is there is no tag specified in your user data. The logs should have some message if that is the cause of the error.

[arnaldo2792]

Yes, you need to provide the tag otherwise host-ctr (our client that runs the bootstrap containers) will fail to pull down the container image. FWIW, you can use this from the admin container to only get the bootstrap container's logs: journalctl -D /.bottlerocket/rootfs/var/log -u [email protected]

You can use bootstrap-containers@<container-name-in-API>.service to filter out logs from the journal 👍

[nike21oct]

hi @stmcginnis and all, i have a EKS cluster which is using bottlerocket AMI and nginx as a ingress controller and when i implemented these IP tables rules by bootstrap-container my application stop open from outside the cluster , i mean my ingress is not functioning my nginx ingress controller pod went to crashloopbackoff , ngix controller load balancer in aws in target group the Protocol : Port is TLS: 32443 and health check is using protocol http and port is 32002, so what should i need to do?
please help me here

[arnaldo2792]

From the CIS benchmark:

With a default accept policy the firewall will accept any packet that is not configured to
be denied. It is easier to allow list acceptable usage than to deny list unacceptable
usage.

Thus, you need to manually configure the rules you need for your application to work.

[nike21oct]

my pod is getting failed with reason of liveness and readiness probe failed and it is because the iptables rule first drop all traffic and when we allow the node port on iptables it wont work and gets failed as my target group health is running on port 32002 and ingress nginx controller is listening on port 32443

[ozmodiar192]

It sounds like you're having a good ol' fashioned iptables issue. The NodePort range is 30000-32767 so you can get any port in there. This article may be helpful: https://kubernetes.io/blog/2023/05/11/nodeport-dynamic-and-static-allocation/#how-can-you-avoid-nodeport-service-port-conflicts

[nike21oct]

hi @ozmodiar192 , I did not understand your point how i can solve my issue . By reading the article it says about the range only but my issue is when I implement iptable rules to do the hardening of my bottlerocket AMI i am facing a issue.

[ozmodiar192]

I'm sorry, I may be misunderstanding your issue. If you are creating services using nodeport and you've only allowed 3 ports within the node port range, it would make sense it's blocking if other ports get created. Your options are to permit individual nodeports and then explicitly select the open nodeports, or allow a specific range of ports. These may be helpful if that's the case, if not I apologize for misunderstanding the issue.
https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
https://www.tkng.io/services/nodeport/

[nike21oct]

hi @ozmodiar192 , No this is not the issue you misunderstood , the issue is I am having EKS cluster running by using AMI bottlerocket and I am doing the CIS benchmarking of bottlerocket AMI by following a document shared by AWS "https://aws.amazon.com/blogs/containers/validating-amazon-eks-optimized-bottlerocket-ami-against-the-cis-benchmark/"
In these thee is a procedure of IP tables rules change and when I implement IP table rules my nginx ingress controller pod went to crashloopback off and I cannot access my application from outside the cluster.nginx as a ingress controller and when i implemented these IP tables rules by bootstrap-container my application stop open from outside the cluster , i mean my ingress is not functioning my nginx ingress controller pod went to crashloopbackoff , ngix controller load balancer in aws in target group the Protocol : Port is TLS: 32443 and health check is using protocol http and port is 32002, so what should i need to do here ?

[ozmodiar192]

I think you're assuming the CIS benchmarks are meant to handle your application traffic and they're not. They're meant to be a foundation on which to add the rules required for your applications. If you could put these rules in and it was easy and secure and no additional work was required, this would be the OOTB ruleset. It's not, because the default deny puts additional work on administrators to configure additional rules to permit their required traffic. Best of luck!