RFE: exclude 1 node from Nemesis (to catch memory leaks and such)

[mykaul]

In a given scenario, I'd like to ensure that Nemesis does not touch one of the nodes for the whole duration of the test (days, I assume).
The goal is that it doesn't get any faults, restarts or whatnot, but we can see its metrics and specifically, ensure it doesn't suffer from memory leaks. Today it's hard to catch them in the dynamic nature of Nemesis runs across random nodes.

@fruch , @roydahan - thoughts?

Migrated to Jira: SCT-20

[roydahan]

We used to have it in the past where node-1 was always excluded from the nemesis selection.
I can't say it helped us with detecting memory leaks and there were few reasons why we removed this limitation:

1. All rolling restart nemesis (intentional and configuration changes that require rolling restarts) must have it.
2. There were some cases we wanted to make sure there is no data integrity issues and when the test always keep one node unharmed it was very hard to trigger these cases.
   There were probably more reasons, these are the ones from top of my head.

I think that a better approach would be to implement a special case for it or a way to exclude one node by mentioning it in the configuration and apply it only for certain cases.
Then, we should implement a post test validation (teardown_validators) that validates the memory consumption.

@mykaul is there a customer issue that triggers this request? (would help to learn more details for the implementation)

[mykaul]

@mykaul is there a customer issue that triggers this request? (would help to learn more details for the implementation)

https://github.com/scylladb/scylla-enterprise/issues/5333 and https://github.com/scylladb/scylla-enterprise/issues/5332

[nyh]

@mykaul is there a customer issue that triggers this request? (would help to learn more details for the implementation)

scylladb/scylla-enterprise#5333 and scylladb/scylla-enterprise#5332

Also in https://github.com/scylladb/scylla-enterprise/issues/5332#issuecomment-2781378122 I noted that trivial workload which just creates and deletes a table by the same name (I did NOT check that the "same name" limitation is important), leaks memory - no reads or writes to the empty table or anything is even necessary.