Skip to content

Latest commit

 

History

History
258 lines (227 loc) · 36 KB

File metadata and controls

258 lines (227 loc) · 36 KB

AWS Load Balancer Controller

AWS Load Balancer controller Helm chart for Kubernetes

TL;DR:

helm repo add eks https://aws.github.io/eks-charts
# If using IAM Roles for service account install as follows -  NOTE: you need to specify both of the chart values `serviceAccount.create=false` and `serviceAccount.name=aws-load-balancer-controller`
helm install aws-load-balancer-controller eks/aws-load-balancer-controller --set clusterName=my-cluster -n kube-system --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller
# If not using IAM Roles for service account
helm install aws-load-balancer-controller eks/aws-load-balancer-controller --set clusterName=my-cluster -n kube-system

Introduction

AWS Load Balancer controller manages the following AWS resources

  • Application Load Balancers to satisfy Kubernetes ingress objects
  • Network Load Balancers to satisfy Kubernetes service objects of type LoadBalancer with appropriate annotations

Security updates

Note: Deployed chart does not receive security updates automatically. You need to manually upgrade to a newer chart.

Node isolation

As a security best practice, we recommend isolating the controller deployment pods to specific node groups which run critical components. The helm chart provides parameters nodeSelector, tolerations and affinity to configure node isolation. For more information, please refer to the guidance here.

Prerequisites

  • Kubernetes >= 1.19
  • IAM permissions
  • Helm v3
  • Optional dependencies
    • cert-manager
    • Prometheus Operator

The controller runs on the worker nodes, so it needs access to the AWS ALB/NLB resources via IAM permissions. The IAM permissions can either be setup via IAM roles for ServiceAccount or can be attached directly to the worker node IAM roles.

Setup IAM for ServiceAccount

  1. Create IAM OIDC provider

    eksctl utils associate-iam-oidc-provider \
        --region <aws-region> \
        --cluster <your-cluster-name> \
        --approve
    
  2. Download IAM policy for the AWS Load Balancer Controller

    curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json
    
  3. Create an IAM policy called AWSLoadBalancerControllerIAMPolicy

    aws iam create-policy \
        --policy-name AWSLoadBalancerControllerIAMPolicy \
        --policy-document file://iam-policy.json
    

    Take note of the policy ARN that is returned

  4. Create a IAM role and ServiceAccount for the Load Balancer controller, use the ARN from the step above

    eksctl create iamserviceaccount \
    --cluster=<cluster-name> \
    --namespace=kube-system \
    --name=aws-load-balancer-controller \
    --attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
    --approve
    

Setup IAM manually

If not setting up IAM for ServiceAccount, apply the IAM policies from the following URL at minimum.

https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/main/docs/install/iam_policy.json

Upgrading from ALB ingress controller

If migrating from ALB ingress controller, grant additional IAM permissions.

Upgrading from AWS Load Balancer controller v2.1.3 and earlier

  • Additional IAM permissions required, ensure you have granted the required IAM permissions.
  • CRDs need to be updated as follows
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"
  • you can run helm upgrade without uninstalling the old chart completely

Installing cert-manager

If you are setting enableCertManager: true you need to have installed cert-manager and it's CRDs before installing this chart; to install cert-manager follow the installation guide.

The controller helm chart requires the cert-manager with apiVersion cert-manager.io/v1.

Set cluster.dnsDomain (default: cluster.local) to the actual DNS domain of your cluster to include the FQDN in requested TLS certificates.

Installing the Prometheus Operator

If you are setting serviceMonitor.enabled: true you need to have installed the Prometheus Operator ServiceMonitor CRD before installing this chart and have the operator running to collect the metrics. The easiest way to do this is to install the kube-prometheus-stack Helm chart using the installation guide.

Installing the Chart

Note: You need to uninstall aws-alb-ingress-controller. Please refer to the upgrade section below before you proceed. Note: Starting chart version 1.4.1, you need to explicitly set clusterSecretsPermissions.allowAllSecrets to true to grant the controller permission to access all secrets for OIDC feature. We recommend configuring access to individual secrets resource separately [link].

Add the EKS repository to Helm:

helm repo add eks https://aws.github.io/eks-charts

Install the TargetGroupBinding CRDs:

kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"

Install the AWS Load Balancer controller, if using iamserviceaccount

# NOTE: The clusterName value must be set either via the values.yaml or the Helm command line. The <k8s-cluster-name> in the command
# below should be replaced with name of your k8s cluster before running it.
helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=<k8s-cluster-name> --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller

Install the AWS Load Balancer controller, if not using iamserviceaccount

helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=<k8s-cluster-name>

Upgrade

The new controller is backwards compatible with the existing ingress objects. However, it will not coexist with the older aws-alb-ingress-controller. The old controller must be uninstalled completely before installing the new version.

Kubectl installation

If you had installed the previous version via kubectl, uninstall as follows

$ kubectl delete deployment -n kube-system alb-ingress-controller
$ kubectl delete clusterRole alb-ingress-controller
$ kubectl delete ClusterRoleBinding alb-ingress-controller
$ kubectl delete ServiceAccount -n kube-system alb-ingress-controller

# Alternatively you can find the version of the controller and delete as follows
$ kubectl describe deployment  -n kube-system  alb-ingress-controller |grep Image
      Image:      docker.io/amazon/aws-alb-ingress-controller:v1.1.8
# You can delete the deployment now
$ kubectl delete deployment -n kube-system alb-ingress-controller
# In this case, the version is v1.1.8, the rbac roles can be removed as follows
$ kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.8/docs/examples/rbac-role.yaml

Helm installation

If you had installed the incubator/aws-alb-ingress-controller Helm chart, uninstall as follows

# NOTE: If installed under a different chart name and namespace, please specify as appropriate
$ helm delete aws-alb-ingress-controller -n kube-system

If you had installed the 0.1.x version of eks-charts/aws-load-balancer-controller chart earlier, the upgrade to chart version 1.0.0 will not work due to incompatibility of the webhook api version, uninstall as follows

$ helm delete aws-load-balancer-controller -n kube-system

Uninstalling the Chart

helm delete aws-load-balancer-controller -n kube-system

If you setup IAM Roles for ServiceAccount, you can cleanup as follows

eksctl delete iamserviceaccount --cluster <cluster-name> --namespace kube-system --name aws-load-balancer-controller

HA configuration

Chart release v1.2.0 and later enables high availability configuration by default.

  • The default number of replicas is 2. You can pass--set replicaCount=1 flag during chart installation to disable this. Due to leader election, only one controller will actively reconcile resources.
  • The default priority class for the controller pods is system-cluster-critical
  • Soft pod anti-affinity is enabled for controller pods with topologyKey: kubernetes.io/hostname if you don't configure custom affinity and set configureDefaultAffinity to true
  • Pod disruption budget (PDB) has not been set by default. If you plan on running at least 2 controller pods, you can pass --set podDisruptionBudget.maxUnavailable=1 flag during chart installation

Configuration

The following tables lists the configurable parameters of the chart and their default values. The default values set by the application itself can be confirmed here.

Parameter Description Default
image.repository image repository public.ecr.aws/eks/aws-load-balancer-controller
image.tag image tag <VERSION>
image.pullPolicy image pull policy IfNotPresent
clusterName Kubernetes cluster name None
cluster.dnsDomain DNS domain of the Kubernetes cluster, included in TLS certificate requests cluster.local
securityContext Set to security context for pod {}
resources Controller pod resource requests & limits {}
priorityClassName Controller pod priority class system-cluster-critical
nodeSelector Node labels for controller pod assignment {}
tolerations Controller pod toleration for taints {}
affinity Affinity for pod assignment {}
configureDefaultAffinity Configure soft pod anti-affinity if custom affinity is not configured true
topologySpreadConstraints Topology Spread Constraints for pod assignment {}
deploymentAnnotations Annotations to add to deployment {}
podAnnotations Annotations to add to each pod {}
podLabels Labels to add to each pod {}
additionalLabels Labels to add to all components {}
rbac.create if true, create and use RBAC resources true
serviceAccount.annotations optional annotations to add to service account None
serviceAccount.automountServiceAccountToken Automount API credentials for a Service Account true
serviceAccount.imagePullSecrets List of image pull secrets to add to the Service Account []
serviceAccount.create If true, create a new service account true
serviceAccount.name Service account to be used None
terminationGracePeriodSeconds Time period for controller pod to do a graceful shutdown 10
ingressClass The ingress class to satisfy alb
createIngressClassResource Create ingressClass resource true
ingressClassParams.name IngressClassParams resource's name, default to the aws load balancer controller's name None
ingressClassParams.create If true, create a new ingressClassParams true
ingressClassParams.spec IngressClassParams defined ingress specifications {}
region The AWS region for the kubernetes cluster None
vpcId The VPC ID for the Kubernetes cluster None
awsApiEndpoints Custom AWS API Endpoints None
awsApiThrottle Custom AWS API throttle settings None
awsMaxRetries Maximum retries for AWS APIs None
defaultTargetType Default target type. Used as the default value of the alb.ingress.kubernetes.io/target-type and service.beta.kubernetes.io/aws-load-balancer-nlb-target-type" annotations.Possible values are ip and instance. instance
enablePodReadinessGateInject If enabled, targetHealth readiness gate will get injected to the pod spec for the matching endpoint pods None
enableShield Enable Shield addon for ALB None
enableWaf Enable WAF addon for ALB None
enableWafv2 Enable WAF V2 addon for ALB None
ingressMaxConcurrentReconciles Maximum number of concurrently running reconcile loops for ingress None
logLevel Set the controller log level - info, debug None
metricsBindAddr The address the metric endpoint binds to ""
webhookBindPort The TCP port the Webhook server binds to None
webhookTLS.caCert TLS CA certificate for webhook (auto-generated if not provided) ""
webhookTLS.cert TLS certificate for webhook (auto-generated if not provided) ""
webhookTLS.key TLS private key for webhook (auto-generated if not provided) ""
webhookNamespaceSelectors Namespace selectors for the wekbook None
keepTLSSecret Reuse existing TLS Secret during chart upgrade true
serviceAnnotations Annotations to be added to the provisioned webhook service resource {}
serviceMaxConcurrentReconciles Maximum number of concurrently running reconcile loops for service None
targetgroupbindingMaxConcurrentReconciles Maximum number of concurrently running reconcile loops for targetGroupBinding None
targetgroupbindingMaxExponentialBackoffDelay Maximum duration of exponential backoff for targetGroupBinding reconcile failures None
syncPeriod Period at which the controller forces the repopulation of its local object stores None
watchNamespace Namespace the controller watches for updates to Kubernetes objects, If empty, all namespaces are watched None
disableIngressClassAnnotation Disables the usage of kubernetes.io/ingress.class annotation None
disableIngressGroupNameAnnotation Disables the usage of alb.ingress.kubernetes.io/group.name annotation None
defaultSSLPolicy Specifies the default SSL policy to use for HTTPS or TLS listeners None
externalManagedTags Specifies the list of tag keys on AWS resources that are managed externally []
livenessProbe Liveness probe settings for the controller (see values.yaml)
env Environment variables to set for aws-load-balancer-controller pod None
hostNetwork If true, use hostNetwork false
dnsPolicy Set dnsPolicy if required ClusterFirst
extraVolumeMounts Extra volume mounts for the pod []
extraVolumes Extra volumes for the pod []
defaultTags Default tags to apply to all AWS resources managed by this controller {}
replicaCount Number of controller pods to run, only one will be active due to leader election 2
podDisruptionBudget Limit the disruption for controller pods. Require at least 2 controller replicas and 3 worker nodes {}
updateStrategy Defines the update strategy for the deployment {}
enableCertManager If enabled, cert-manager issues the webhook certificates instead of the helm template, requires cert-manager and it's CRDs to be installed false
enableEndpointSlices If enabled, controller uses k8s EndpointSlices instead of Endpoints for IP targets false
enableBackendSecurityGroup If enabled, controller uses shared security group for backend traffic true
backendSecurityGroup Backend security group to use instead of auto created one if the feature is enabled ``
disableRestrictedSecurityGroupRules If disabled, controller will not specify port range restriction in the backend security group rules false
objectSelector.matchExpressions Webhook configuration to select specific pods by specifying the expression to be matched None
objectSelector.matchLabels Webhook configuration to select specific pods by specifying the key value label pair to be matched None
serviceMonitor.enabled Specifies whether a service monitor should be created, requires the ServiceMonitor CRD to be installed false
serviceMonitor.additionalLabels Labels to add to the service account {}
serviceMonitor.interval Prometheus scrape interval 1m
serviceMonitor.namespace Namespace in which Prometheus is running None
clusterSecretsPermissions.allowAllSecrets If true, controller has access to all secrets in the cluster. false
controllerConfig.featureGates set of key: value pairs that describe AWS load balance controller features {}
ingressClassConfig.default If true, the ingressclass will be the default class of the cluster. false
enableServiceMutatorWebhook If false, disable the Service Mutator webhook which makes all new services of type LoadBalancer reconciled by the lb controller true