Nginx Gateway time out 504

[ArkShocer]

Hi,
I have set up Terrakube on a K8s Cluster on Azure. Currently, after accessing the UI and creating my first organization, I get an cors error displayed in the console. Does anyone know what could be the reason for cors to throw those errors? The login with sso for azure ad went without any problems.

Error 1:

Access to XMLHttpRequest at 'https://terrakube-api.aks.example.dev/api/v1/organization' from origin 'https://terrakube-ui.aks.example.dev' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Error 2: requests.js:1

POST https://terrakube-api.aks.example.dev/api/v1/organization net::ERR_FAILED 504 (Gateway Timeout)

[alfespa17]

I think that could be an issue with your ingress configuration that is returning 504 gateway timeout, maybe you could check your ingress logs

The CORS configuration can be checked inside the API logs when the application is starting like the following:

[ArkShocer]

These are the api logs, and the pod goes healthy after the init sequence.

024-03-06 15:57:21.130  INFO 1 --- [           main] o.t.a.p.scheduler.module.CacheService    : Run module index scan
2024-03-06 15:57:21.135  INFO 1 --- [           main] o.t.a.p.scheduler.module.CacheService    : Create Schedule Module Refresh: DEFAULT.TerrakubeV2_ModuleRefresh_a4ab83c0-adbe-4051-b65d-f3996d37b1ed
2024-03-06 15:57:21.233  INFO 1 --- [           main] o.t.a.p.scheduler.module.CacheService    : Disable Old Module Refresh
2024-03-06 15:57:21.317  INFO 1 --- [           main] o.t.a.p.scheduler.module.CacheService    : jobDetail is null false
2024-03-06 15:57:21.317  INFO 1 --- [           main] o.t.a.p.scheduler.module.CacheService    : Delete Old Quartz Job
2024-03-06 15:57:21.401 ERROR 1 --- [           main] o.t.a.p.scheduler.module.CacheService    : Couldn't obtain triggers for job: No record found for selection of Trigger with key: 'DEFAULT.TerrakubeV2_ModuleRefresh' and statement: SELECT * FROM QRTZ_CRON_TRIGGERS WHERE SCHED_NAME = 'schedulerFactoryBean' AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ?
2024-03-06 15:57:21.881  WARN 1 --- [           main] .s.s.UserDetailsServiceAutoConfiguration :
Using generated security password: NotThisTimeExample:)
This generated password is for development use only. Your security configuration must be updated before running your application in production.
2024-03-06 15:57:22.007  INFO 1 --- [           main] o.t.a.p.s.a.dex.DexWebSecurityAdapter    : Loading CORS https://terrakube-ui.aks.example.dev
2024-03-06 15:57:22.033  INFO 1 --- [           main] o.s.s.web.DefaultSecurityFilterChain     : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@8a641ed, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@7ffe0471, org.springframework.security.web.context.SecurityContextPersistenceFilter@2f75c3f3, org.springframework.security.web.header.HeaderWriterFilter@6914bb4a, org.springframework.web.filter.CorsFilter@1694aadb, org.springframework.security.web.csrf.CsrfFilter@31ef9ac6, org.springframework.security.web.authentication.logout.LogoutFilter@2b749404, org.springframework.security.oauth2.server.resource.web.BearerTokenAuthenticationFilter@4d548ae7, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@287cfa9c, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@10decc1e, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@16d9c898, org.springframework.security.web.session.SessionManagementFilter@1a7084ec, org.springframework.security.web.access.ExceptionTranslationFilter@f3745b2, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@739174f1]
2024-03-06 15:57:25.967  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
2024-03-06 15:57:26.057  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2024-03-06 15:57:26.136  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class java.util.Date
2024-03-06 15:57:26.231  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class java.sql.Date
2024-03-06 15:57:26.233  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class java.sql.Time
2024-03-06 15:57:26.233  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class java.sql.Timestamp
2024-03-06 15:57:26.233  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class java.time.Instant
2024-03-06 15:57:26.233  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class java.time.OffsetDateTime
2024-03-06 15:57:26.233  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class java.util.TimeZone
2024-03-06 15:57:26.234  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class java.net.URL
2024-03-06 15:57:26.247  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Day
2024-03-06 15:57:26.249  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Hour
2024-03-06 15:57:26.251  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.ISOWeek
2024-03-06 15:57:26.252  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Minute
2024-03-06 15:57:26.258  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-03-06 15:57:26.258  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2024-03-06 15:57:26.260  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Month
2024-03-06 15:57:26.261  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 2 ms
2024-03-06 15:57:26.268  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Quarter
2024-03-06 15:57:26.269  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Second
2024-03-06 15:57:26.270  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Time
2024-03-06 15:57:26.271  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Week
2024-03-06 15:57:26.272  INFO 1 --- [           main] com.yahoo.elide.Elide                    : Registering serde for type : class com.yahoo.elide.datastores.aggregation.timegrains.Year
2024-03-06 15:57:27.189  INFO 1 --- [           main] org.terrakube.api.ServerApplication      : Started ServerApplication in 19.325 seconds (JVM running for 20.221)
2024-03-06 15:58:21.053  INFO 1 --- [_MisfireHandler] o.s.s.quartz.LocalDataSourceJobStore     : Handling 1 trigger(s) that missed their scheduled fire-time.
2024-03-06 15:59:21.010  INFO 1 --- [_MisfireHandler] o.s.s.quartz.LocalDataSourceJobStore     : Handling 1 trigger(s) that missed their scheduled fire-time.
2024-03-06 15:59:22.057  INFO 1 --- [_ClusterManager] o.s.s.quartz.LocalDataSourceJobStore     : ClusterManager: detected 1 failed or restarted instances.
2024-03-06 15:59:22.057  INFO 1 --- [_ClusterManager] o.s.s.quartz.LocalDataSourceJobStore     : ClusterManager: Scanning for instance "terrakube-api-6595c67bf-9jxmd1709740518301"'s failed in-progress jobs.
2024-03-06 16:00:21.008  INFO 1 --- [_MisfireHandler] o.s.s.quartz.LocalDataSourceJobStore     : Handling 1 trigger(s) that missed their scheduled fire-time.

[ArkShocer]

Here are the logs from the ingress controller:

The first 13 Messages are all the requests that get sent to the api from my button clicks in the gui and the ones after the space are the answers from the timeout I think

10.100.144.10 - - [06/Mar/2024:17:20:36 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 87 0.008 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.007 200 580ecf48fe40f095ec77358c678d4e64
10.100.144.10 - - [06/Mar/2024:17:20:45 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.002 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.002 200 fb01df35068533ff5c7beafc03b2093d
10.100.144.10 - - [06/Mar/2024:17:20:45 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.007 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.008 200 805673299da29b316cfcea7ab296d50e
10.100.144.10 - - [06/Mar/2024:17:20:46 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.007 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.007 200 5421331c20b44e6dd6330abb57f4051d
10.100.144.10 - - [06/Mar/2024:17:20:46 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.003 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.003 200 e01aaebe5f6bb6105f0e2b37799186a8
10.100.144.10 - - [06/Mar/2024:17:20:46 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.001 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.002 200 096f9767e3100ce11c16c36aaaf5094f
10.100.144.10 - - [06/Mar/2024:17:20:46 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.002 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.002 200 c582e9b4789fd7cabdbff2eecd9e394d
10.100.144.10 - - [06/Mar/2024:17:20:46 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.002 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.001 200 e3cb157e36a6ae7a4425f94a3cbc5c67
10.100.144.10 - - [06/Mar/2024:17:20:46 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.019 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.019 200 e324a8a5ccbb0d1d7f16e0c58f68cb3d
10.100.144.10 - - [06/Mar/2024:17:20:46 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.007 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.006 200 557fc8203104ef5c397712677c751d15
10.100.144.10 - - [06/Mar/2024:17:20:47 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.011 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.011 200 06f1a2d2282287ca9012e15a176b54e4
10.100.144.10 - - [06/Mar/2024:17:20:47 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.014 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.014 200 d7ef0c57c8062415c4f51e14d1c2abf3
10.100.144.10 - - [06/Mar/2024:17:20:47 +0000] "OPTIONS /api/v1/organization HTTP/2.0" 200 0 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 40 0.008 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 0.008 200 cdf4f7e3ae335eb07243a595f64b061a

2024/03/06 17:21:23 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "GET /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:31 +0000] "GET /api/v1/applications/nx-aks-infra-terrakube/syncwindows?name=nx-aks-infra-terrakube&appNamespace=argocd HTTP/2.0" 200 40 "https://completelysecret.aks.example.dev/applications/argocd/nx-aks-infra-terrakube?view=tree&orphaned=false&resource=" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 91 0.019 [argocd-argocd-server-80] [] 10.100.144.83:8080 40 0.019 200 f863ea6adf588434c68bf5783e94f1d3
2024/03/06 17:21:36 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:36 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 119 60.000 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.001 504 65469d206e04ad75608e1c30d5857822
2024/03/06 17:21:45 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:45 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 60.000 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.000 504 87b8956847d789088a9cb2762323f698
2024/03/06 17:21:45 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:45 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 60.001 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.000 504 b7258b6d0b318879ee0b0f2a9dfe7d41
2024/03/06 17:21:46 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:46 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 60.000 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.000 504 42f4c4bd48b0a83ff3773dc4f154a8ec
2024/03/06 17:21:46 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:46 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 59.999 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.000 504 6a434a24f966b98d4a36623f7026bdaf
2024/03/06 17:21:46 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:46 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 59.999 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.000 504 ba9c19548d3fd977fb0917b00e00f1f0
2024/03/06 17:21:46 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:46 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 60.001 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.000 504 6affae5973b954783feba6820e54e933
2024/03/06 17:21:46 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:46 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 60.000 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.000 504 2bec44641c5e7c551eb825c652364145
2024/03/06 17:21:46 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:46 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 60.000 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.000 504 af0d9fab556f27a17262739b2b69092b
2024/03/06 17:21:46 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:46 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 60.000 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.001 504 3f9d6f6b7d1d015bf7dd56781ca34bf1
2024/03/06 17:21:47 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:47 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 60.001 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.001 504 b469f4788a7ac056057b67b390c0be1f
2024/03/06 17:21:47 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:47 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 59.999 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.000 504 0762993d47df57dc27686f1c67756f99
2024/03/06 17:21:47 [error] 5344#5344: *11917505 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.100.144.10, server: terrakube-api.aks.example.dev, request: "POST /api/v1/organization HTTP/2.0", upstream: "http://10.100.144.104:8080/api/v1/organization", host: "terrakube-api.aks.example.dev", referrer: "https://terrakube-ui.aks.example.dev/"
10.100.144.10 - - [06/Mar/2024:17:21:47 +0000] "POST /api/v1/organization HTTP/2.0" 504 562 "https://terrakube-ui.aks.example.dev/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" 97 60.000 [nx-aks-infra-terrakube-terrakube-api-service-8080] [] 10.100.144.104:8080 0 60.000 504 810670f4a43471e0d1f78adb673c84d6

[alfespa17]

Try doing a port forward to the API like the following

user@pop-os:~$ curl -v "http://localhost:8080/api/v1/organization"
*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /api/v1/organization HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< Set-Cookie: JSESSIONID=BAA3A9409D4D06D46C33EC12F97EB3D1; Path=/; HttpOnly
< WWW-Authenticate: Bearer
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Length: 0
< Date: Wed, 06 Mar 2024 20:57:08 GMT
< 
* Connection #0 to host localhost left intact

It should return http 401 because your are not authenticated, maybe you can try something similar from another namespace

[ArkShocer]

Doing a port forward and then the curl command results in the following:

PS C:\Users\nx> curl -v "http://localhost:8080/api/v1/organization"
*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> GET /api/v1/organization HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 401
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< Set-Cookie: JSESSIONID=4640FFA3A59175AFFC20ED3D8E7D6BC3; Path=/; HttpOnly
< WWW-Authenticate: Bearer
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Length: 0
< Date: Thu, 07 Mar 2024 09:05:38 GMT
<
* Connection #0 to host localhost left intact

and this one from within the k8s (ingress namespace to terrakube namespace)

nx-aks-infra-ingress-nginx-controller-6455d45b8c-tgb2w:/etc/nginx$ curl -v "http://10.100.144.104:8080/api/v1/organization"
*   Trying 10.100.144.104:8080...
* Connected to 10.100.144.104 (10.100.144.104) port 8080
> GET /api/v1/organization HTTP/1.1
> Host: 10.100.144.104:8080
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 401
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< Set-Cookie: JSESSIONID=3ACE007276BC20D92697871EBF113DBA; Path=/; HttpOnly
< WWW-Authenticate: Bearer
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Length: 0
< Date: Thu, 07 Mar 2024 09:20:52 GMT
<
* Connection #0 to host 10.100.144.104 left intact

[alfespa17]

It looks like the API is working fine becuase it is returning http 401, maybe you can try adding some annotation to change the ingress timeout for your nginx.

https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#custom-timeouts

nginx.ingress.kubernetes.io/proxy-connect-timeout
nginx.ingress.kubernetes.io/proxy-send-timeout
nginx.ingress.kubernetes.io/proxy-read-timeout

I hope that can help you

[ArkShocer]

I set those on the ingress of the api with 120 timeout and other than waiting longer for the response it changed nothing. Then I tried it with the ingress of the ui and it also sadly changed nothing :(