Archive node panic

[ivanlei]

Describe the bug

Strangelove reports 1 of 3 archive nodes is panic'ing while coming back up after an unrelated crash on the box.

Hey agoric team!  Any of you seen this before?  We can restore it from our last snapshot if needed, but curious if you had ideas on if this can be solved a different way.
1:25 PM INF agd delegating to JS executable args=["ag-chain-cosmos", "--home", "/home/operator/.agoric", "start", "--home", "/home/operator/cosmos"] binary=/usr/src/agoric-sdk/packages/cosmic-swingset/src/entrypoint.js
1:25 PM INF Removing intrinsics.Symbol.dispose
1:25 PM INF Removing intrinsics.Symbol.asyncDispose
1:31 PM INF ABCI client type: committing
1:31 PM INF starting node with ABCI Tendermint in-process
1:31 PM INF service start impl=multiAppConn module=proxy msg={}
1:31 PM INF service start connection=query impl=committingClient module=abci-client msg={}
1:31 PM INF service start connection=snapshot impl=committingClient module=abci-client msg={}
1:31 PM INF service start connection=mempool impl=committingClient module=abci-client msg={}
1:31 PM INF service start connection=consensus impl=committingClient module=abci-client msg={}
1:31 PM INF service start impl=EventBus module=events msg={}
1:31 PM INF service start impl=PubSub module=pubsub msg={}
1:31 PM INF service start impl=IndexerService module=txindex msg={}
1:31 PM INF ABCI Handshake App Info hash="�I�A|�!,��������y�nɊ��=\x1eE������" height=15774978 module=consensus protocol-version=0 software-version=0.35.0-u15.0
1:31 PM INF ABCI Replay Blocks appHeight=15774978 module=consensus stateHeight=15774981 storeHeight=15774981
1:31 PM INF Applying block height=15774979 module=consensus
1:31 PM INF minted coins from module account amount=8589996ubld from=mint module=x/bank
1:31 PM INF Loading slog sender modules: @agoric/telemetry/src/flight-recorder.js
1:31 PM INF Launching SwingSet kernel
1:32 PM INF Launched SwingSet kernel
1:32 PM ERR portHandler threw (Error#1)
1:32 PM ERR Error#1: Unimplemented reset state from 15774981 to 15774979

panic: cannot initialize Controller: Error: Unimplemented reset state from 15774981 to 15774979

goroutine 438 [running]:
github.com/Agoric/agoric-sdk/golang/cosmos/app.(*GaiaApp).initController(_, {{0x7f48aab8e680, 0xc000064038}, {0x7f48aab9e990, 0xc2a98ccac0}, {{0xb, 0x0}, {0xc2a97e8ff8, 0x8}, 0xf0b503, ...}, ...}, ...)
    /usr/src/agoric-sdk/golang/cosmos/app/app.go:971 +0x59d
github.com/Agoric/agoric-sdk/golang/cosmos/app.(*GaiaApp).ensureControllerInited(...)
    /usr/src/agoric-sdk/golang/cosmos/app/app.go:996
github.com/Agoric/agoric-sdk/golang/cosmos/x/swingset.AppModule.BeginBlock({{}, {{0x7f48aab75dc0, 0xc03ff09200}, {0x7f48aaba3e38, 0xc00069e030}, {{0x7f48aab9dce0, 0xc00069e030}, 0xc0006b4000, {0x7f48aab75dc0, 0xc03ff09170}, ...}, ...}, ...}, ...)
    /usr/src/agoric-sdk/golang/cosmos/x/swingset/module.go:136 +0x55
github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x7f48aab8e680, 0xc000064038}, {0x7f48aab9e990, 0xc2a98ccac0}, {{0xb, 0x0}, {0xc2a97e8ff8, 0x8}, 0xf0b503, ...}, ...}, ...)
    /go/pkg/mod/github.com/agoric-labs/[email protected]/types/module/module.go:484 +0x1bb
github.com/Agoric/agoric-sdk/golang/cosmos/app.(*GaiaApp).BeginBlocker(...)
    /usr/src/agoric-sdk/golang/cosmos/app/app.go:1001
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc2a98c2200, 0x20, 0x20}, {{0xb, 0x0}, {0xc2a97e8ff8, 0x8}, 0xf0b503, {0x444c2bf, ...}, ...}, ...})
    /go/pkg/mod/github.com/agoric-labs/[email protected]/baseapp/abci.go:183 +0x843
github.com/tendermint/tendermint/abci/client.(*committingClient).BeginBlockSync(_, {{0xc2a98c2200, 0x20, 0x20}, {{0xb, 0x0}, {0xc2a97e8ff8, 0x8}, 0xf0b503, {0x444c2bf, ...}, ...}, ...})
    /go/pkg/mod/github.com/agoric-labs/[email protected]/abci/client/committing_client.go:332 +0x1a4
github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc2a98c2200, 0x20, 0x20}, {{0xb, 0x0}, {0xc2a97e8ff8, 0x8}, 0xf0b503, {0x444c2bf, ...}, ...}, ...})
    /go/pkg/mod/github.com/agoric-labs/[email protected]/proxy/app_conn.go:81 +0x55
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x7f48aab8f7c8?, 0xc02e8a2060}, {0x7f48aab975f0, 0xc292ac6a40}, 0xc00122f860, {0x7f48aab9f358, 0xc045024180}, 0xc038603a00?)
    /go/pkg/mod/github.com/agoric-labs/[email protected]/state/execution.go:307 +0x51d
github.com/tendermint/tendermint/state.ExecCommitBlock({0x7f48aab975f0, 0xc292ac6a40}, 0xc00122f860, {0x7f48aab8f7c8, 0xc02e8a2060}, {0x7f48aab9f358?, 0xc045024180?}, 0x7f48a8248fc5?)
    /go/pkg/mod/github.com/agoric-labs/[email protected]/state/execution.go:537 +0x67
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlocks(_, {_, _}, {{{0xb, 0x0}, {0xc2929d4710, 0x8}}, {0xc2929d4718, 0x8}, 0x204855, ...}, ...)
    /go/pkg/mod/github.com/agoric-labs/[email protected]/consensus/replay.go:495 +0x2a5
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocksWithContext(_, {_, _}, {{{0xb, 0x0}, {0xc2929d4710, 0x8}}, {0xc2929d4718, 0x8}, 0x204855, ...}, ...)
    /go/pkg/mod/github.com/agoric-labs/[email protected]/consensus/replay.go:410 +0x486
github.com/tendermint/tendermint/consensus.(*Handshaker).HandshakeWithContext(0xc03ff59c20, {0x7f48aab8e680, 0xc000064050}, {0x7f48aaba07d8?, 0xc03bbe8340?})
    /go/pkg/mod/github.com/agoric-labs/[email protected]/consensus/replay.go:274 +0x405
github.com/tendermint/tendermint/node.doHandshake({_, _}, {_, _}, {{{0xb, 0x0}, {0xc2929d4710, 0x8}}, {0xc2929d4718, 0x8}, ...}, ...)
    /go/pkg/mod/github.com/agoric-labs/[email protected]/node/node.go:330 +0x1c8
github.com/tendermint/tendermint/node.NewNodeWithContext({0x7f48aab8e680, 0xc000064050}, 0xc0006d17c0, {0x7f48aab899a0, 0xc0014a85a0}, 0xc04e39f100, {0x7f48aab6ed60, 0xc2661f81b0}, 0x7f47af4c5ae0?, 0x7f48aab63300, ...)
    /go/pkg/mod/github.com/agoric-labs/[email protected]/node/node.go:797 +0x5b7
github.com/tendermint/tendermint/node.NewNode(0x0?, {0x7f48aab899a0?, 0xc0014a85a0?}, 0x0?, {0x7f48aab6ed60?, 0xc2661f81b0?}, 0x1?, 0x7f48abdea0a0?, 0x0?, {0x7f48aab8f7c8, ...}, ...)
    /go/pkg/mod/github.com/agoric-labs/[email protected]/node/node.go:719 +0xbb
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x7f48aabab158, 0xc001c58f00}, 0x0, {0x0, 0x0}, {0x7f48aaba3e38, ...}, ...}, ...)
    /go/pkg/mod/github.com/agoric-labs/[email protected]/server/start.go:314 +0x769
github.com/cosmos/cosmos-sdk/server.StartCmd.func2.2()
    /go/pkg/mod/github.com/agoric-labs/[email protected]/server/start.go:149 +0x58
github.com/cosmos/cosmos-sdk/server.wrapCPUProfile.func2()
    /go/pkg/mod/github.com/agoric-labs/[email protected]/server/start.go:560 +0x29
created by github.com/cosmos/cosmos-sdk/server.wrapCPUProfile
    /go/pkg/mod/github.com/agoric-labs/[email protected]/server/start.go:559 +0x24a

[ivanlei]

Coming from

agoric-sdk/packages/cosmic-swingset/src/launch-chain.js

Line 765 in 0006934

`Unimplemented reset state from ${savedHeight} to ${blockHeight}`,

// we're being asked to rewind by more than one block, or execute something
// more than one block in the future, neither of which we can accommodate.
// Keep throwing forever.
decohered = Error(
  // TODO unimplemented
  `Unimplemented reset state from ${savedHeight} to ${blockHeight}`,
);

[ivanlei]

This is very likely another instance of #6736

[warner]

yep, dupe of #6736