Error on Node Deployment for Penumbra Example in Cosmos-Operator

[danbryan]

Description:

While attempting to deploy the Penumbra example provided in the Cosmos-Operator documentation, specifically using the configuration found here, I encountered an issue during the node startup process.

Deployment Details:

• Operator: Cosmos-Operator
• Chain: Penumbra
• Error Details: The node fails to start due to errors related to deprecated configuration options and EOF errors during the ABCI handshake. Below is the log output from the node:

Logs from Fresh Sync

I[2024-10-28|19:26:04.153] deprecated usage found in configuration file usage="[fastsync] table detected. This section has been renamed to [blocksync]. The values in this deprecated section will be disregarded." 
I[2024-10-28|19:26:04.153] deprecated usage found in configuration file usage="fast_sync key detected. This key has been renamed to block_sync. The value of this deprecated key will be disregarded."   
I[2024-10-28|19:26:04.162] service start module=proxy msg="Starting multiAppConn service" impl=multiAppConn                                                               
I[2024-10-28|19:26:04.162] service start module=abci-client connection=query msg="Starting socketClient service" impl=socketClient                                        
I[2024-10-28|19:26:04.162] service start module=abci-client connection=snapshot msg="Starting socketClient service" impl=socketClient                                     
I[2024-10-28|19:26:04.162] service start module=abci-client connection=mempool msg="Starting socketClient service" impl=socketClient                                      
I[2024-10-28|19:26:04.162] service start module=abci-client connection=consensus msg="Starting socketClient service" impl=socketClient                                    
I[2024-10-28|19:26:04.162] service start module=events msg="Starting EventBus service" impl=EventBus                                                                     
I[2024-10-28|19:26:04.162] service start module=pubsub msg="Starting PubSub service" impl=PubSub                                                                         
I[2024-10-28|19:26:04.164] service start module=txindex msg="Starting IndexerService service" impl=IndexerService                                                         
I[2024-10-28|19:26:04.165] ABCI Handshake App Info module=consensus height=0 hash=0000000000000000000000000000000000000000000000000000000000000000 software-version=0.80.0 protocol 
I[2024-10-28|19:26:04.165] ABCI Replay Blocks module=consensus appHeight=0 storeHeight=0 stateHeight=0                                                                    
E[2024-10-28|19:26:04.165] Stopping abci.socketClient for error: read message: EOF module=abci-client connection=consensus                                                                                
I[2024-10-28|19:26:04.165] service stop module=abci-client connection=consensus msg="Stopping socketClient service" impl=socketClient                                   
E[2024-10-28|19:26:04.165] consensus connection terminated. Did the application crash? Please restart CometBFT module=proxy err="read message: EOF"                                                      
ERROR: failed to create node: error during handshake: error on replay: read message: EOF                                                                                                                 
Stream closed EOF for penumbra-1/penumbra-mainnet-sentry-0 (node) 

here are the PD logs

2024-10-28T19:36:44.154227Z  INFO tower_abci::v037::server: listening for requests
2024-10-28T19:36:44.154342Z  INFO tower_abci::v037::server: listening for requests
2024-10-28T19:36:44.154424Z  INFO tower_abci::v037::server: listening for requests
2024-10-28T19:36:44.154515Z  INFO tower_abci::v037::server: listening for requests
2024-10-28T19:36:44.156511Z  INFO abci:Info: penumbra_app::server::info: reporting height in info query info=Info { version: "0.37.9", block_version: 11, p2p_version: 8, abci_version: "1.0.0" } state_
thread 'tokio-runtime-worker' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-abci-0.11.1/src/v037/server.rs:179:75:
called `Result::unwrap()` on an `Err` value: ActorTerminated 

When doing a snapshot restore from polkachu, i get an app hash. Tested with ghcr.io/strangelove-ventures/heighliner/penumbra:v0.80.0.

Issue Summary:

• Configuration Warnings: The log indicates the use of deprecated configuration options (fastsync vs. blocksync).
• ABCI Handshake Error: The node fails to complete the ABCI handshake and terminates with an EOF error, preventing the node from fully starting.
• Node Termination: The connection to the consensus module fails, resulting in a failure to create the node.

Expected Behavior:

• The node should start successfully without errors.

Steps to Reproduce:

1. Deploy the Penumbra example as described in the Cosmos-Operator documentation.
2. Observe the logs from the node deployment process.

Request:

• Please update the example configuration to deploy a working penumbra node in the cosmos-operator. Ideally a solution that works for full sync, and snapshot restore.
• Any guidance or suggestions for a temporary workaround would be appreciated in the mean time.