[Critical Bug Fix] TTL timer has possibility to lose efficacy for ElasticSearch storage in all <= 9.2.0(6.x/7.x/8.x included) releases

[wu-sheng]

Phenomena

1. The expired indices had never been released before rebooting the OAP cluster or scaling the cluster size to 1. Found the following logs in all OAP pod logs, The selected first getAddress is xxx.xxx.xx.xx:port. The remove stage is skipped.
2. Can't reboot the OAP pod and see the endless rolling logs table: xxx does not exist. OAP is running in 'no-init' mode, waiting... retry 3s later.

Root cause

This bug exists for many years, back to Oct. 2018. TTL timer expected queryRemoteNodes always returns a certain ordered OAP instance list, which makes one OAP node would be selected to take the responsibility of removing expired indices and create the latest(today's) indices when rolling.

But, typically and proved, when using k8s coordinator, the k8s coordinator would not return an order instance list, which could have no OAP nodes selected, and the TTL timer would not really work in any case.
In this case, most indices could be created normally as new telemetry data would trigger index creation automatically. But

1. Expired indices would not be removed
2. Some features(as not be used) may not have new telemetry data, so, no latest date index was created. But the rebooting would verify and expect the latest data index, which could lead to Phenomena <2>.

Fix

The pull request to fix this is #9632.

[huailide]

Version/image:skywalking-oap-server:8.9.1
We use k8S to deploy skywalking. Not using Helm 3 as described in the official website document, but use the Docker image（https://hub.docker.com/r/apache/skywalking-oap-server） to deploy oap-server(version 8.9.1, show in 4th attachment info02.png).
At the beginning, we encountered this problem. By using the configuration（K8S_UID） shown in the 1st attachment(config.png), it seems that TTL can work normally(show in the 2nd and 3rd attachment info01.png，info01-02.png).I'm not sure it's reasonable for me to do this, but it really solved my problem.

[wu-sheng]

I think K8S_UID would not fix this. This is just for assigning a new ID.
You could check the official fix.

[huailide]

Thanks for the reply.
For our company, it is hard to upgrade the oap service rapidly. So, I'm glad it(By giving k8s metadata.uid to ENV SW_CLUSTER_K8S_UID) solved my problem.
I'm just sharing my experience and the upgrade for the oap is already planned.

[wu-sheng]

You may get this running just because of luck(randomly ordered correctly). I don't ask you to upgrade. I just warn you, every single change on your images or env, would make this failure. This is not a thing could be fixed from this way.

[huailide]

Thx for the replay. You're right
I got the idea(Injecting pod metadata.uid into oap server.) by referring to this document
https://blog.csdn.net/stomach/article/details/116156157
At that time, we couldn't find a second plan.

[henrikrexed]

Hi,

i'm using the latest version of Apache Skywalking.
and i'm having the same exact issue.
how can i resolve this?

[wu-sheng]

This should have been fixed. Which cluster coordinator are you using? Which IP are registered in your coordinator server?