Removal of preserve_client_ip generates inconsistent security group rules

[knisbet]

Describe the bug
When the preserve_client_ip.enabled=true annotation is removed from a service, the generated Security Group configuration for an EKS cluster will mismatch the configuration of the AWS NLB resulting in blocked traffic.

Steps to reproduce

1. Create a k8s service with the following annotations

    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: {{ .Values.certArn }}
    service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
    service.beta.kubernetes.io/aws-load-balancer-alpn-policy: "HTTP2Preferred"

2. Add the preserve_client_ip annotation

    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true

3. Rollback the change to the service by removing the preserve_client_ip.enabled=true annotation
4. Observe that the configured NLB remains in preserve_client_ip mode
5. Observe that the node security-group rules are generated as if preserve_client_ip is disabled

Expected outcome
The EKS security group to match the configuration of the NLB that is created/managed by the aws-load-balancer-controller.

Environment

• AWS Load Balancer controller version: v2.10.0
• Kubernetes version: v1.29.8-eks-a737599
• Using EKS (yes/no), if so version? yes, eks.13

Additional Context:
I reviewed #2705 which does indicate the controller is expected to ignore annotation that have been removed. However, that ticket didn't go into the behaviour of the EKS security group upon removal of the annotation.

In our case we had an outage occur because we introduced the preservice_client_ip=true as part of a helm chart and release of our software. This release had an unrelated issue and when we rolled back to the previous helm chart we encountered an outage due to the Security Group blocking all external traffic to the service.

This outage occurred because upon removal of the annotation, the NLB stayed in client IP preservation mode, but the Security Group Rules applied to the nodes were generated with client IP preservation disabled as per: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.10/guide/service/nlb/#worker-node-security-groups-rules

We'll be taking steps to avoid this, but our team wanted to report the issue as we doubt we'll be the only ones who make a mistake like this by rolling back a configuration change and expecting the previous behaviour to resume.