Graceful shutdowns for IRaftHttpCluster leaders?

[Arkensor]

Hi,

I was wondering how a leader in an IRaftHttpCluster could shut down gracefully.

My requirement is, that if an instance is asked to shut down and its currently the cluster leader, that others will take over for him and he will be known to have left the cluster.

I tested calling RemoveMemberAsync during the IClusterMemberLifetime.OnStop event. It appeared to go through and returned true but the other members in the cluster appear to get no information about the leader "leaving" and only move on as soon as the heart beats start to fail.

Is there an intended way for how to handle this? Is it even possible to leave the cluster on a CTRL+C / close process scenario?
Or does the library not handle this / was not designed to do this?

Thank you :)

[sakno]

Graceful shutdown is an operation that must be initiated externally by cluster admin. It consists of the following steps:

1. Stops the node
2. Informs the cluster that the stopped node is not a member of the cluster

The last operation is a regular Raft operation that must be accepted by the leader and committed by the majority of nodes. RemoveMemberAsync should be called on leader node only.

[Arkensor]

I remember reading a specification about this in the raft paper saying something along the lines of "if the leader wants to remove himself from the cluster, he must first replicate him leaving to a majority and then step down"

Does this library follow that? Or do I need to kill the leader and remove him as "unreachable" member once a new one has been elected?

[sakno]

Yes, it follows:

dotNext/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.Membership.cs

Lines 162 to 166 in ebda0df

	// local member is removed, downgrade it
	var newState = new StandbyState(this);
	using var currentState = state;
	state = newState;
	}

[sakno]

The removed node moves to standby state, which is an extension to Raft. Standby node never tracks election timeout so it cannot be moved to Candidate state. It means that such node will never be elected as a leader.