Timing issue with validator set submissions

[jkilpatr]

 [2021-04-11T17:48:51Z ERROR relayer::valset_relaying] Valset cost estimate for Nonce 124 failed with Err(EthereumRestError(JsonRpcError { code: 3, message: "execution reverted: Validator signature does not match.", data: "Some(String(\"0x08c379a00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002356616c696461746f72207369676e617475726520646f6573206e6f74206d617463682e0000000000000000000000000000000000000000000000000000000000\"))" }))

You may see this in test failures at some points. It's not a bug in the tests it's an actual timing bug in the code. Somehow it's possible for the Orchestrator and chain to enter a state where a successfully and correctly signed validator set is not submitted to the Ethereum chain. Instead an earlier validator set is submitted, constantly failing because it's not the correct value.

I've been chasing this one for a while and the only thing I've really confirmed is that the signatures are in fact correct, at no point is the signature itself wrong. This means it has to be an ordering/timing issue.