Ingress ALB

[sagarnitd]

Please, provide ingress.yaml for AWS Internal ALB

[alfespa17]

There is one example using EKS in the ingress configurations here, not sure if that can help you.

https://github.com/AzBuilder/terrakube-helm-chart/blob/main/examples/AzureAuthentication-Example4.md

[sagarnitd]

@alfespa17
Can we have Amazon EKS + Load Balancer (ALB) ?

[sagarnitd]

Can we connect sometime? @alfespa17.
I will make documentation for AWS EKS + ALB.
I have tried many times to setup Terrakube on aws eks but still failing.

[alfespa17]

Not sure if I will be able to help I dont work with AWS but as far I now the only things that needs to be update are the ingress annotation.

I mostly use Azure and GCP, maybe you can check the discussions or other issues in the main repository maybe someone there could be able to help.

[genedev22]

@sagarnitd Any luck on this?

I installed the AWS load balancer controller. Briefly: I used a Helm chart to install the controller and used eksctl to configure the IRSA (IAM role + cluster service account) for it.

I then tried to follow the ingress setup shown in https://github.com/AzBuilder/terrakube-helm-chart/blob/main/examples/AzureAuthentication-Example4.md.

The best I was able to do from that was get four network load balancers to deploy. The four NLBs corresponded to the API, Executor, Registry and UI services. Furthermore, the listeners in each NLB were not listening on port 443 as I hoped; they were listening on the cluster service ports themselves.

Although I'm still troubleshooting things, I made some progress:

I disabled the individual ingresses in values.yaml as such.

## Ingress properties
ingress:
  useTls: false
  includeTlsHosts: false
  ui:
    enabled: false
  api:
    enabled: false
  registry:
    enabled: false
  dex:
    enabled: false

I don't know if it matters, but I disabled the TLS (useTLS: false) since the plan is to have the services behind the ALB with a wildcard TLS certificate in ACM:

Then I deployed a separate ALB ingress that maps to the API, Dex (not Executor) Registry, and UI services. The path settings are from copying over the paths in the example. I had trouble using pathType: Prefix, but pathType: ImplementationSpecific worked for me.

Also: In values.yaml, I still have the services as ClusterIP.

Here's my ingress definition in a separate YAML file.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: alb-ingress
  namespace: terrakube
  annotations:
    alb.ingress.kubernetes.io/load-balancer-name: terrakube-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:<account id>:certificate/<id of your SSL cert in ACM>
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/group.name: alb-group
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - host: terrakube-ui.<snip>.com
      http:
        paths:
          - path: "/*"
            pathType: ImplementationSpecific
            backend:
              service:
                name: terrakube-ui-service
                port:
                  number: 8080
    - host: terrakube-api.<snip>.com
      http:
        paths:
          - path: "/dex/*"
            pathType: ImplementationSpecific
            backend:
              service:
                name: terrakube-dex
                port:
                  number: 5556
          - path: "/*"
            pathType: ImplementationSpecific
            backend:
              service:
                name: terrakube-api-service
                port:
                  number: 8080
    - host: terrakube-reg.<snip>.com
      http:
        paths:
          - path: "/*"
            pathType: ImplementationSpecific
            backend:
              service:
                name: terrakube-registry-service
                port:
                  number: 8075

Finally, applied it after deploying the Helm chart: kubectl apply -f alb-ingress.yaml -n terrakube.

I get one ALB deployed and the listeners have targets that seem to line up with the service ports. The UI login page resolves for me and I can log into Cognito (the IdP for the Dex connector I am using).

I don't think this last part will be fully relevant to you, but you might still encounter errors after deploying the separate ingress. In my case, this happened:

After authenticating with Cognito, I am getting this peculiar error during the Dex redirect:

Bad Request
Unregistered redirect_uri ("http://terrakube-ui.minikube.net").

It's probably an error specific to my configuration, but my values.yaml has no reference to http://terrakube-ui.minikube.net. I'll be opening another issue about it.

[genedev22]

@sagarnitd @alfespa17

I made more progress and I'm now able to log into the application inside my EKS cluster:

• With my Cognito IDP Dex connector.
• Domain name with HTTPS.
• And the single ALB load balancer that holds the load balancer rules per subdomain host and satisfies the health check settings per path.

Maybe either of you would be interested in seeing these notes, so I'll leave these notes here:

Right now I'm using these Ingress settings inside values.yaml along side my separate ALB Ingress definition:

## Ingress properties
ingress:
  useTls: true
  includeTlsHosts: true
  ui:
    enabled: false
    domain: "terrakube-ui.<snip>.com"
    ingressClassName: "alb"
    tlsSecretName: tls-secret-ui-terrakube
    annotations: {}
  api:
    enabled: false
    domain: "terrakube-api.<snip>.com"
    ingressClassName: "alb"
    tlsSecretName: tls-secret-api-terrakube
    annotations: {}
  registry:
    enabled: false
    domain: "terrakube-reg.<snip>.com"
    ingressClassName: "alb"
    tlsSecretName: tls-secret-reg-terrakube
    annotations: {}
  dex:
    enabled: false
    annotations: {}

The ALB load balancer ingress specification above allows me to finally log into the UI. The only issue I had at that point was that the provisioned load balancer was not automatically applying a health check based on each pod's Readiness probe settings; I wanted to figure out how to get the ALB load balancer controller to apply the health check settings automatically.

Unfortunately, I could not figure out how to apply custom health check paths and health check ports inside one Ingress definition for the ALB. However, I can define the custom health check configurations if I go back to using multiple ingress definitions; then I can apply separate annotations (as values.yaml allows me to do) for each ingress.

These are the ALB health check annotations that work if applied to each ingress separately:

annotations:
    alb.ingress.kubernetes.io/healthcheck-path: <health check path>
    alb.ingress.kubernetes.io/healthcheck-port: <health check port>

But oddly enough when I use separate ingresses, the load balancer no longer satisfies the priority between rules under the Terrakube API service; I specifically have trouble logging into the UI unless the load balancer /dex/* path is prioritized over the more general /* path under the terrakube-api.my-domain.com host:

- host: terrakube-api.<snip>.com
  http:
	paths:
	  - path: "/dex/*"
		pathType: ImplementationSpecific
		backend:
		  service:
			name: terrakube-dex
			port:
			  number: 5556
	  - path: "/*"
		pathType: ImplementationSpecific
		backend:
		  service:
			name: terrakube-api-service
			port:
			  number: 8080

And this is in spite of using these annotations for the ALB load balancer to try to force the dex Ingress before the api Ingress:

alb.ingress.kubernetes.io/priority: <priority number>
alb.ingress.kubernetes.io/group.name: alb-group
alb.ingress.kubernetes.io/load-balancer-name: terrakube-alb

Finally, what did work to get one ALB, with the healthcheck settings per path, and maintain rule priorities in the API subdomain host was:

1. The values.yaml ingress specification, as you helped me with.
2. The separate ALB definition (shown in my previous message), which maintains /dex/* path prioritized over /* under terrakube-api.my-domain.com host.
3. Editing the Helm chart service definition files templates/service-api.yaml, templates/service-registry.yaml, and charts/dex/templates.values.yaml to add the ALB healthcheck annotations, respectively:

annotations:
  alb.ingress.kubernetes.io/healthcheck-path: /actuator/health/readiness
  alb.ingress.kubernetes.io/healthcheck-port: '8080'

annotations:
  alb.ingress.kubernetes.io/healthcheck-path: /actuator/health/readiness
  alb.ingress.kubernetes.io/healthcheck-port: '8075'

annotations:
  alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready  
  alb.ingress.kubernetes.io/healthcheck-port: '5558'

I suspect I am overlooking something regarding the ALB controller that would make the load balancer provisioning a bit cleaner, but what I have shown seems to give me everything for my DNS/Load balancer settings.

[alfespa17]

hello @genedev22 I think this comment can help you.

#146 (comment)

DEX and the API are two different components but to simplify the deployment we just reuse the same API domain for DEX in one specific path.

You can deploy those 2 services using a different domain if you want.

[alfespa17]

You could in theory simply deploy DEX using its own helm chart in another domain and simple change your terrakube helm values to disable the internal DEX that terrakube provides and simply use the other DEX like:

dex:
  config:
    issuer: https://my-custom-dex.with-other-domain.com

[genedev22]

@alfespa17 I like your idea. This should allow me to get around the load balancer priority rules not getting satisfied (as I had encountered) and let me bring back the UI/API/Registry ingress settings back into values.yaml.

Given that I'm ready to try out the application, I'll defer that separate DEX resource exploration for now, but thank you for the suggestion.

[BenjaminDecreusefond]

Hi @genedev22 !

Can you show your full values.yml please ? I'm running into the same issues as you and I'm having trouble to fix my UI ! I'm not getting through the login page 😢

[genedev22]

Hi @BenjaminDecreusefond,

Quick note: I have a consistent issue with my Executor pod crashing during startup, but a quick kubectl delete pod <pod id> had fixed it. Try deleting your API pod perhaps if you haven't done so? Same goes for any other pod whose logs suggest a startup problem.

I started off using these examples as a reference.

• https://docs.terrakube.io/getting-started/user-management/amazon-cognito
• https://github.com/AzBuilder/terrakube-helm-chart/blob/main/examples/CognitoAuthentication-Example1.md

My values.yaml does not have the individual ingresses enabled. I deployed a separate yaml file for one ingress that covers all of the routes for my ALB. I happen to be using a standalone Postgres Aurora DB, but your mileage may vary. I happen to be using the default Redis resource because ElastiCache requires SSL connectivity (and to my understanding, the Terrakube Helm chart does not make that yet available).

If you see an error in the UI, try opening your browser's tools (F12) and check if it's trying to go an https or http endpoint. In my case, I have it setup for https.

The devs can speak better than me on this, but my crude understanding of the auth workflow (based on my configuration) is:

• Go to the login page, say at https://terrakube-ui.your-domain.com.
• Get directed to https://terrakube-api.your-domain.com/dex, and then in turn to your IdP login page (in my case, Cognito).
• From there, the rest of the OIDC workflow follows. To add a little more detail:
  • After successful login (into your IdP), you'll see a consent screen under the "Dex" UI. After accepting, you'll then see the Terrakube UI present you with choosing or creating an "organization".
  • I only have an authorization code grant for the OAuth grant type. No custom scopes (just email, openid, and profile). In general: Make sure settings match between values.yaml and your app client settings in your IdP.

Other: The subdomains don't have to match the defaults shown as far as I can tell, so you should have flexibility in renaming them ... but make sure you're consistent with your subdomains in values.yaml and wherever your Ingress and DNS settings are applied.

## Global Name
name: "terrakube"

global:
  imagePullSecrets: []
  
security:
  useOpenLDAP: false
  adminGroup: "<name of your AWS Cognito admin group>"
  patSecret: "<PAT secret in base64>"
  internalSecret: "<internal secret in base64>"
  dexClientId: cognito-app
  dexClientScope: "email openid profile offline_access groups"
  dexIssuerUri: "https://terrakube-api.your-domain.com/dex"
  existingSecret: false

## Dex
dex:
  enabled: true
  existingSecret: false
  config:
    issuer: "https://terrakube-api.your-domain.com/dex"
    storage:
      type: memory
    web:
      http: 0.0.0.0:5556
      allowedOrigins: ['*']
      skipApprovalScreen: true
    oauth2:
      responseTypes: ["code", "token", "id_token"]
    connectors:
    - type: oidc
      id: cognito
      name: AWS Cognito
      config:
        issuer: "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_<user pool id>"
        clientID: "<your AWS Cognito app's client id>"
        clientSecret: "<your AWS Cognito app's secret key>"
        redirectURI: "https://terrakube-api.your-domain.com/dex/callback"
        scopes:
          - openid
          - email
          - profile
        insecureSkipEmailVerified: true
        insecureEnableGroups: true
        userNameKey: "cognito:username"
        claimMapping:
          groups: "cognito:groups"
    staticClients:
    - id: cognito-app
      redirectURIs:
      - 'https://terrakube-ui.your-domain.com'
      - '/device/callback'
      - 'http://localhost:10000/login'
      - 'http://localhost:10001/login'
      name: 'cognito-app'
      public: true

## Terraform Storage
storage:
  defaultStorage: false
  aws:
    accessKey: "<the IAM access key for an IAM key pair>"
    secretKey: "<the IAM secret key for an IAM key pair>"
    bucketName: "name-of-bucket-in-your-aws-account"
    region: "your-aws-region"

# Default Redis Configuration
redis:
  architecture: "standalone"
  auth:
    enabled: true
    password: "your-redis-password"

## API properties
api:
  existingSecret: false
  enabled: true
  image: azbuilder/api-server
  version: ""
  replicaCount: "1"
  serviceType: "ClusterIP"
  serviceAccountName: ""
  rbac:
    create: false
    roleName: "terrakube-api-role"
    roleBindingName: "terrakube-api-role-binding"
  secrets:
   - terrakube-api-secrets
  resources: {}
  podLabels: {}
  defaultDatabase: false
  defaultRedis: true
  loadSampleData: false
  terraformReleasesUrl: "https://releases.hashicorp.com/terraform/index.json"
  securityContext: {}
  containerSecurityContext: {}
  imagePullSecrets: []
  initContainers: []
  cache:
    moduleCacheMaxTotal: "128"
    moduleCacheMaxIdle: "128"
    moduleCacheMinIdle: "64"
    moduleCacheTimeout: "600000"
    moduleCacheSchedule: "0 */3 * ? * *"
  properties:
    databaseType: "POSTGRESQL"
    databaseHostname: "<your postgres hostname>"
    databaseName: "terrakube"
    databaseUser: "terrakube"
    databaseSchema: "public"
    databasePassword: "<your DB password>"
    databaseSslMode: "require"
    databasePort: "5432"
    redisHostname: ""
    redisPassword: ""

## The database port is only used for mysql databases
## SslMode values are disable, allow, prefer, require, verify-ca, verify-full. Default mode is "disable".
## Reference: https://jdbc.postgresql.org/documentation/publicapi/org/postgresql/PGProperty.html#SSL_MODE

## Executor properties
executor:
  existingSecret: false
  enabled: true
  image: "azbuilder/executor"
  version: ""
  replicaCount: "1"
  serviceType: "ClusterIP"
  serviceAccountName: ""
  secrets:
   - terrakube-executor-secrets
  resources: {}
  podLabels: {}
  podAnnotations: {}
  apiServiceUrl: "http://terrakube-api-service:8080"
  properties:
    toolsRepository: "https://github.com/AzBuilder/terrakube-extensions"
    toolsBranch: "main"
  securityContext: {}
  containerSecurityContext: {}
  imagePullSecrets: []
  initContainers: []
  
## Registry properties
registry:
  enabled: true
  existingSecret: false
  image: azbuilder/open-registry
  version: ""
  replicaCount: "1"
  serviceType: "ClusterIP"
  serviceAccountName: ""
  secrets:
   - terrakube-registry-secrets
  resources: {}
  podLabels: {}
  securityContext: {}
  containerSecurityContext: {}
  imagePullSecrets: []
  initContainers: []
  
## UI Properties
ui:
  enabled: true
  existingSecret: false
  image: azbuilder/terrakube-ui
  version: ""
  replicaCount: "1"
  serviceType: "ClusterIP"
  serviceAccountName: ""
  resources: {}
  podLabels: {}
  securityContext: {}
  containerSecurityContext: {}
  imagePullSecrets: []
  initContainers: []
  
## Ingress properties
ingress:
  useTls: true
  includeTlsHosts: true
  ui:
    enabled: false
    domain: "terrakube-ui.your-domain.com"
    ingressClassName: "alb"
    tlsSecretName: tls-secret-ui-terrakube
    annotations: {}
  api:
    enabled: false
    domain: "terrakube-api.your-domain.com"
    ingressClassName: "alb"
    tlsSecretName: tls-secret-api-terrakube
    annotations: {}
  registry:
    enabled: false
    domain: "terrakube-reg.your-domain.com"
    ingressClassName: "alb"
    tlsSecretName: tls-secret-reg-terrakube
    annotations: {}
  dex:
    enabled: false
    annotations: {}

and this is my one Ingress. I don't claim this is 'the' way to setup your Ingress; it's just what finally worked for me ... such that it gave me one ALB, the correct mapping of hosts and paths to the Terrakube services. In my case: plus adding ALB health check annotations in the service definition yaml files so the ALB health checks were reaching out to the correct health check paths and correct health check ports.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: alb-ingress
  namespace: terrakube
  annotations:
    alb.ingress.kubernetes.io/load-balancer-name: terrakube-alb
    alb.ingress.kubernetes.io/group.name: alb-group
    
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:<your account id>:certificate/<your ACM certificate id>
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/target-type: ip

    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '15'  
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '5'  
    alb.ingress.kubernetes.io/healthy-threshold-count: '2'  
    alb.ingress.kubernetes.io/unhealthy-threshold-count: '2' 
spec:
  ingressClassName: alb
  rules:
    - host: terrakube-ui.your-domain.com
      http:
        paths:
          - path: "/*"
            pathType: ImplementationSpecific
            backend:
              service:
                name: terrakube-ui-service
                port:
                  number: 8080
    - host: terrakube-api.your-domain.com
      http:
        paths:
          - path: "/dex/*"
            pathType: ImplementationSpecific
            backend:
              service:
                name: terrakube-dex
                port:
                  number: 5556
          - path: "/*"
            pathType: ImplementationSpecific
            backend:
              service:
                name: terrakube-api-service
                port:
                  number: 8080
    - host: terrakube-reg.your-domain.com
      http:
        paths:
          - path: "/*"
            pathType: ImplementationSpecific
            backend:
              service:
                name: terrakube-registry-service
                port:
                  number: 8075