Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing permission in docs #3917

Open
derek-howard opened this issue Oct 29, 2024 · 2 comments
Open

Missing permission in docs #3917

derek-howard opened this issue Oct 29, 2024 · 2 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@derek-howard
Copy link

Describe the problem
There seems to be a missing permission that is required for the AWS load balancer controller to function properly, however, I have deployed the exact same AWS load balancer controller helm chart, with the same values, to other AWS accounts and never encountered this issue. Same Terraform code as well. So, I am lost here.

ec2:GetSecurityGroupsForVpc

Steps to reproduce
Deploy Terraform code for base infrastructure, including EKS.

Deploy AWS load balancer controller pod identity, with the permissions here

Deploy the AWS load balancer controller helm chart, v1.9.0

Deploy ArgoCD helm chart. Target groups get provisioned just fine, but fails to create the ALB.

Expected outcome
Successful provisioning of ALB

Environment

  • AWS Load Balancer controller version: 1.9.0
  • Kubernetes version: 1.31
  • Using EKS (yes/no), if so version? Yes, 1.31

Additional Context:

Reconciler error","controller":"ingress","object":{"name":"core"},"namespace":"","name":"core","reconcileID":"","error":"operation error Elastic Load Balancing v2: CreateLoadBalancer, https response error StatusCode: 403, RequestID: cd7592df-8a02-41bb-8d69-d86aa57b8996, api error AccessDenied: User: arn:aws:sts::000000000000:assumed-role/aws-lbc-main/eks-main-aws-load-b-d4a16fdd-3080-4fd8-8356-90f21bb153b8 is not authorized to perform: ec2:GetSecurityGroupsForVpc

@zschorn
Copy link

zschorn commented Oct 29, 2024

Just some additional information, same for controller version 1.8.1 and 1.9.2.
I can confirm this setup was working for several clusters before October.
A google search showed not much, but perhaps this helps:
AWS Managed Policy: ElasticLoadBalancingFullAccess

https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ElasticLoadBalancingFullAccess.html

Current version v8 contains the right for "GetSecurityGroupsForVpc"
but version v7 does not.
This policy was updated 4 days ago, October 24, 2024, 22:21 UTC
https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ElasticLoadBalancingFullAccess.html
It seems AWS changed some required rights, but I could not find any announcements or release notes regarding this change.
This lead to some not so nice complications.
Documentation from AWS for the load balancer controller here, and the documentation in git is not updated.
https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.9.0/docs/install/iam_policy.json
https://docs.aws.amazon.com/eks/latest/userguide/lbc-helm.html

If anyone also can find an announcement for this change by AWS, it would be great.
So it is not directly an error in the LBC, but some missing communication from AWS.
Hope this helps

@zac-nixon
Copy link
Contributor

zac-nixon commented Oct 29, 2024

Can you share which region(s) you are encountering this? The ELB team has rolled back a deployment yesterday.

I just talked with the ELB team, and this is unexpected behavior. For anyone else that runs into this issue you can add ec2:GetSecurityGroupsForVpc to your controller's IAM policy.

Here's the PR to augment the IAM policy: https://github.com/kubernetes-sigs/aws-load-balancer-controller/pull/3921/files

@shraddhabang shraddhabang added the kind/bug Categorizes issue or PR as related to a bug. label Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

4 participants