Failed to release lock: `Operation cannot be fulfilled on leases.coordination.k8s.io "skupper-site-leader"`

[vsomwanshi]

Describe the bug
We have OpenShift cluster A where we have only one namespace xyz and we call it as our main site where we issue token and link that token with the multiple namespaces in Openshift Cluster B ( roughly we have around 40 namespaces in cluster B ).

When we initially deployed skupper with HA in Cluster A under namespaces xyz it was working fine later at some point the skupper-routers from OpenShift cluster B started restarting with an error Failed to release lock: Operation cannot be fulfilled on leases.coordination.k8s.io "skupper-site-leader": the object has been modified; please apply your changes to the latest version and try again.

Below is the logs from one of the namespaces isd-br-dev from cluster B.

025/08/13 15:58:42 INFO updating network status info component=kube.flow.statusSync configmap=skupper-network-status
E0813 15:58:43.047087 1 leaderelection.go:436] error retrieving resource lock isd-br-dev/skupper-site-leader: Get "https://172.30.0.1:443/apis/coordination.k8s.io/v1/namespaces/isd-br-dev/leases/skupper-site-leader": context deadline exceeded
I0813 15:58:43.047126 1 leaderelection.go:297] failed to renew lease isd-br-dev/skupper-site-leader: context deadline exceeded
E0813 15:58:43.053052 1 leaderelection.go:322] Failed to release lock: Operation cannot be fulfilled on leases.coordination.k8s.io "skupper-site-leader": the object has been modified; please apply your changes to the latest version and try again
2025/08/13 15:58:43 COLLECTOR: Lost leader lock after 15.920086783s 

Just because of these frequent restarts connection with microservices is failing.

How To Reproduce

Deploy skupper in HA and link with multiple namespaces.

Expected behavior
Skupper sites should be able to identify the leader in ha and flow the traffic accordingly.

Environment details

• Skupper CLI: 2.1.0
• Skupper Operator (if applicable): [e.g. 1.5.0, 1.4.3]
• Platform: OpenShift

Additional context
Add any other context about the problem here.

[vsomwanshi]

@c-kruse This issue is not related to #2248. Both are completely different issues.

#2248 - This issue is related to the skupper-router pod on the site where we issue the token and link access is enabled ( skupper-router is not setup in HA mode )

#2250 - This issue is related to the skupper-router pods on a different site where we redeem the token to create a link when we deploy a skupper-router in HA on the site where we issue a token. Below is the diagram for reference.

We noticed below error in Site X, Y, Z when skupper-router is deployed in HA mode in Site A

025/08/13 15:58:42 INFO updating network status info component=kube.flow.statusSync configmap=skupper-network-status
E0813 15:58:43.047087 1 leaderelection.go:436] error retrieving resource lock isd-br-dev/skupper-site-leader: Get "https://172.30.0.1:443/apis/coordination.k8s.io/v1/namespaces/isd-br-dev/leases/skupper-site-leader": context deadline exceeded
I0813 15:58:43.047126 1 leaderelection.go:297] failed to renew lease isd-br-dev/skupper-site-leader: context deadline exceeded
E0813 15:58:43.053052 1 leaderelection.go:322] Failed to release lock: Operation cannot be fulfilled on leases.coordination.k8s.io "skupper-site-leader": the object has been modified; please apply your changes to the latest version and try again
2025/08/13 15:58:43 COLLECTOR: Lost leader lock after 15.920086783s 

[c-kruse]

Thanks for the diagram! I suspect that you could be misinterpreting the errors logged: the locks in question on Sites X, Y and Z that are failing have nothing to do with having a peer site that is set up in HA mode. I suspect it could have more to do with the cluster X, Y and Z are on or the Sites themselves.

A few occurrences of this error should be okay, and shouldn't necessarily disrupt operation in any noticeable way. In short what has happened is this: The "adaptor" container running as a side car to the router takes out and holds a lock through the kube API to say that it is the one adaptor in the Site/namespace that is keeping track of resource status. It usually has no competition for the lock except for when the Pod gets rescheduled/restarted and more than one Pod is momentarily up at a time or when it is IN an HA Site. When something goes wrong with that Lock the adaptor quits and restarts (I forget why - but I remember we had some reason for this.)

With that in mind, I've got a few questions:

1. Could you quantify about how often this happens, and describe what happens to the skupper-router Pod when it does? I would expect that the adaptor container gets restarted while the router container is left to happily hum along.
2. Are you seeing other cluster wide impacts to stability? (this kube api reqeust is timing out - perhaps others are too.)
3. Are Sites X, Y, Z, HA sites themselves? Any other reason there could be multiple replicas of the router pods?

[vsomwanshi]

1. Could you quantify about how often this happens, and describe what happens to the skupper-router Pod when it does? I would expect that the adaptor container gets restarted while the router container is left to happily hum along.
   --> This happens very frequently, skupper-router pods keep restarting as per the liveness or readiness defined in the configuration. Yes, it is the kube-adaptor container which keeps restarting.

2. Are you seeing other cluster wide impacts to stability? (this kube api reqeust is timing out - perhaps others are too.)
   --> Yes, it is a cluster wide impact as i have already shown in the diagram, Site X, Y, Z are the individual namespaces which are hosted in single OpenShift Cluster. Site A is a namespace hosted in separate OpenShift cluster which has skupper-router deployed in HA.

3. Are Sites X, Y, Z, HA sites themselves? Any other reason there could be multiple replicas of the router pods?
   --> It is already answered in question 2. Site X,Y,Z are not in HA. They have single skupper-router pods which are linked with Site A which has 2 skupper-router pods. Reason behind having the skupper-router in HA in Site A because it the namespace where we have all the frontend services defined which would connecting to all the Site X,Y,Z which has backend services. We have 1:n architecture so Site A would be quiet busy always in initiating TLS connections. Hence we are planning to deploy HA in Site A

[c-kruse]

Thanks I'll bring this to the team and see if anyone has ideas for how we can help.

For question 2 - I meant to ask if your cluster openshift control plane is in a degraded state. What I'm seeing looks a whole lot like what I might expect to see were etcd on the brink of falling over.

[vsomwanshi]

Thanks I'll bring this to the team and see if anyone has ideas for how we can help.

For question 2 - I meant to ask if your cluster openshift control plane is in a degraded state. What I'm seeing looks a whole lot like what I might expect to see were etcd on the brink of falling over.

Our cluster is completely healthy as well as we have enough compute available in cluster

[vsomwanshi]

@c-kruse Do we have any inputs on this ? As moving forward in production will have more than 40 sites connecting to site A and single skupper-router pod would not be able to handle the TLS connections so we would need at-least 2 skupper-router pods in site A. I will set this up again and monitor it for some time in development environment and share the logs for reference.