Problem: the gravity nodes in the integration tests sometimes have a consensus failure

[farazsth98]

I noticed this when repeatedly running my own integration tests, or the happy_path_test.go integration test. Randomly during certain test runs, the gravity0 to gravity4 validator containers will panic and exit, with their logs stating:

5:05PM ERR CONSENSUS FAILURE!!! err="kv store with key <nil> has not been registered in stores" module=consensus stack="goroutine 177 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:727 +0x4c\npanic({0x1e3d1e0, 0xc00059de50})\n\t/usr/local/go/src/runtime/panic.go:884 +0x212\ngithub.com/cosmos/cosmos-sdk/store/cachemulti.Store.GetKVStore({{0x2c5eba0, 0xc000534e80}, 0xc000f7e510, 0xc001368c90, {0x0, 0x0}, 0x0, 0xc001368cc0}, {0x0, 0x0})\n\t/go/pkg/mod/github.com/cosmos/[email protected]/store/cachemulti/store.go:196 +0x10a\ngithub.com/cosmos/cosmos-sdk/types.Context.KVStore({{0x2c4fd70, 0xc00012c000}, {0x2c608e0, 0xc0005352c0}, {{0xb, 0x0}, {0xc004a24610, 0xc}, 0x12, {0xcba7b68, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/types/context.go:261 +0x7b\ngithub.com/peggyjv/gravity-bridge/module/v2/x/gravity/keeper.Keeper.setLastUnbondingBlockHeight({{0x0, 0x0}, {0x0, 0x0}, {{0x0, 0x0}, 0x0, {0x0, 0x0}, {0x0, ...}, ...}, ...}, ...)\n\t/go/src/github.com/PeggyJV/gravity-bridge/module/x/gravity/keeper/keeper.go:117 +0x67\ngithub.com/peggyjv/gravity-bridge/module/v2/x/gravity/keeper.Hooks.AfterValidatorBeginUnbonding(...)\n\t/go/src/github.com/PeggyJV/gravity-bridge/module/x/gravity/keeper/hooks.go:25\ngithub.com/cosmos/cosmos-sdk/x/staking/types.MultiStakingHooks.AfterValidatorBeginUnbonding({_, _, _}, {{0x2c4fd70, 0xc00012c000}, {0x2c608e0, 0xc0005352c0}, {{0xb, 0x0}, {0xc004a24610, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/staking/types/hooks.go:55 +0x116\ngithub.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.AfterValidatorBeginUnbonding(...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/staking/keeper/hooks.go:46\ngithub.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.beginUnbondingValidator({{0x2c383c8, 0xc0001a58a0}, {0x2c5eb30, 0xc0003db9c0}, {0x2c53fc0, 0xc0011952c0}, {0x7f7a70c1f7d8, 0xc0005dcb00}, {0x2c5e778, 0xc0013663a8}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/staking/keeper/val_state_change.go:335 +0x6d9\ngithub.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.bondedToUnbonding({{0x2c383c8, 0xc0001a58a0}, {0x2c5eb30, 0xc0003db9c0}, {0x2c53fc0, 0xc0011952c0}, {0x7f7a70c1f7d8, 0xc0005dcb00}, {0x2c5e778, 0xc0013663a8}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/staking/keeper/val_state_change.go:231 +0x178\ngithub.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.ApplyAndReturnValidatorSetUpdates({{0x2c383c8, 0xc0001a58a0}, {0x2c5eb30, 0xc0003db9c0}, {0x2c53fc0, 0xc0011952c0}, {0x7f7a70c1f7d8, 0xc0005dcb00}, {0x2c5e778, 0xc0013663a8}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/staking/keeper/val_state_change.go:192 +0x1198\ngithub.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.BlockValidatorUpdates({{0x2c383c8, 0xc0001a58a0}, {0x2c5eb30, 0xc0003db9c0}, {0x2c53fc0, 0xc0011952c0}, {0x7f7a70c1f7d8, 0xc0005dcb00}, {0x2c5e778, 0xc0013663a8}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/staking/keeper/val_state_change.go:27 +0x7d\ngithub.com/cosmos/cosmos-sdk/x/staking.EndBlocker({{0x2c4fd70, 0xc00012c000}, {0x2c608e0, 0xc0005352c0}, {{0xb, 0x0}, {0xc004a24610, 0xc}, 0x12, {0xcba7b68, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/staking/abci.go:26 +0xff\ngithub.com/cosmos/cosmos-sdk/x/staking.AppModule.EndBlock(...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/x/staking/module.go:172\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).EndBlock(_, {{0x2c4fd70, 0xc00012c000}, {0x2c608e0, 0xc0005352c0}, {{0xb, 0x0}, {0xc004a24610, 0xc}, 0x12, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/[email protected]/types/module/module.go:505 +0x482\ngithub.com/peggyjv/gravity-bridge/module/v2/app.(*Gravity).EndBlocker(_, {{0x2c4fd70, 0xc00012c000}, {0x2c608e0, 0xc0005352c0}, {{0xb, 0x0}, {0xc004a24610, 0xc}, 0x12, ...}, ...}, ...)\n\t/go/src/github.com/PeggyJV/gravity-bridge/module/app/app.go:627 +0x78\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc00156c540, {0x0?})\n\t/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:206 +0x165\ngithub.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc0001214a0, {0xc0001214a0?})\n\t/go/pkg/mod/github.com/tendermint/[email protected]/abci/client/local_client.go:288 +0xdf\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc000394ea0?, {0xc00446a240?})\n\t/go/pkg/mod/github.com/tendermint/[email protected]/proxy/app_conn.go:89 +0x24\ngithub.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2c50e80?, 0xc00132e180}, {0x2c586f0, 0xc00028f6b0}, 0xc0015123c0, {0x2c5fd58, 0xc0004fa168}, 0x11?)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:327 +0x55f\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0014e20b0, 0x7}}, {0xc0014e20c0, 0xc}, 0x1, 0x11, {{0xc0048fd2a0, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/state/execution.go:140 +0x171\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc000f12e00, 0x12)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1659 +0xafd\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc000f12e00, 0x12)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1568 +0x2ff\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1503 +0x94\ngithub.com/tendermint/tendermint/consensus.(*State).enterCommit(0xc000f12e00, 0x12, 0x0)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1541 +0xccf\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc000f12e00, 0xc004429900, {0xc00130b8c0, 0x28})\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:2155 +0x18dc\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc000f12e00, 0xc004429900, {0xc00130b8c0?, 0xc000d81200?})\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:1953 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc000f12e00, {{0x2c30c40?, 0xc0049a0908?}, {0xc00130b8c0?, 0x0?}})\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:856 +0x170\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc000f12e00, 0x0)\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:763 +0x3f9\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/go/pkg/mod/github.com/tendermint/[email protected]/consensus/state.go:379 +0x12d\n"

I can't seem to figure out why this happens, and it's non-deterministic. Just thought I'd raise an issue here in case you guys want to look into that.