UI is failing and UI Pod showing error for Health Check

[Jatin3708]

Hi Team,

I have deployed helm chart on EKS cluster and data stored in AWS RDS, last week it was working fine and i was able to access URL.

Suddenly this week UI is not accessible and in UI pod getting below error :

2025-01-02 10:14:15,147 com.linecorp.armeria.client.endpoint.healthcheck.HttpHealthChecker 232 [armeria-common-worker-epoll-2-2] WARN  [] - [creqId=7767fba2, chanId=7a251c61, laddr=100.64.126.
81:38326, raddr=skywalking-skywalking-helm-oap/172.20.23.161:12800][http://skywalking-skywalking-helm-oap:12800/healthcheck#HEAD] Unexpected 4xx health check response: [EOS, :status=404, conte
nt-type=text/plain; charset=utf-8, server=Armeria/1.29.3, date=Thu, 2 Jan 2025 10:14:15 GMT, content-length=35] A 4xx response generally indicates a misconfiguration of the client. Did you hap
pen to forget to configure the HealthCheckedEndpointGroup's client options?

No customization are done in provided helm chart.

Below are the version used :

set {
name = "oap.image.tag"
value = "10.1.0 "
}
set {
name = "oap.storageType"
value = "postgresql"
}
set {
name = "ui.image.tag"
value = "10.1.0"
}

What can cause this error ?

[wu-sheng]

You should check oap logs. It seems crashed.

[Jatin3708]

@wu-sheng : OAP logs showing no errors.

[wu-sheng]

But the http response code says otherwise.

[Jatin3708]

Hi @wu-sheng : You are absolutely right with your understanding.

But Something has been happened in past few days which has caused this failure. If I use oap and ui image tag as 10.0.0 then this error is not coming.

Further I have fully redeployed the whole helm code in new cluster as well existing cluster(after removing whole stuff) but facing same error. No network policy, nothing has been applied.

This was working perfectly fine before few days with no error.

[wu-sheng]

Maybe, your Elasticsearch is using 10.0 indices/templates but the codes updated? Did you try to erase all index templates and indices?