Destoyed node was chosen as target node for decommission

[juliayakovlev]

In the tests below we run terminate and replace nemesis and then decommissioning of 3 nodes, one by one. I noticed that less than 3 nodes were decommissioned. The reason that previously terminated node was chosen for decommissioning. See examples from log in "Discussion" tab:

https://argus.scylladb.com/tests/scylla-cluster-tests/c7f51531-0a18-4499-9153-fad31baec19f
Destoyed node was chosen as target node for decommission:

< t:2024-12-05 01:32:06,712 f:cluster.py      l:1237 c:sdcm.cluster         p:INFO  > Node perf-latency-nemesis-ubuntu-db-node-c7f51531-1 
[98.84.120.246 | 10.12.2.226] destroyed

< t:2024-12-05 02:44:37,496 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-12-05 02:44:37.494: (InfoEvent Severity.NORMAL) period_type=not-set event_id=c776fe21-9f61-4529-bef8-7592bd7ea177: message=StartEvent - 
ShrinkCluster started decommissioning a node Node perf-latency-nemesis-ubuntu-db-node-c7f51531-1 [98.84.120.246 | 10.12.2.226]

< t:2024-12-05 02:51:49,400 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-12-05 02:51:49.385: (NodetoolEvent Severity.NORMAL) period_type=begin event_id=f0bbdb59-6b85-479a-b6ef-5b0f413a85a8: 
nodetool_command=decommission node=perf-latency-nemesis-ubuntu-db-node-c7f51531-1

https://argus.scylladb.com/tests/scylla-cluster-tests/73f09e4d-0bde-47ed-a0d2-3094210a99bc

Another runs with same problem:
https://argus.scylladb.com/tests/scylla-cluster-tests/41d3fe6b-e837-46df-b528-42fe34f51ca6
https://argus.scylladb.com/tests/scylla-cluster-tests/a3e64331-4a89-475b-9717-dc8f026733f5

[juliayakovlev]

@aleksbykov
May it be the result of #9393?

The commit #9393 was not backported to branch-perf-v15 branch.
When I ran the tests on the branch-perf-v15 (not on master), the issue is not reproduced