Core node doesn't stream blocks to new Bridge node

[jim-sf1]

Summary of Bug

Deployed a new Bridge node on v0.21.5-mocha. Connected it to a fully synched, arhcive, core node on v.3.3.0-mocha (and also tried v3.3.1-mocha) using updated connectivity strings core.ip and core.port. Bridge node throws errors about not finding the correct tendermint service and falls into restart loop. No errors reported on the Core node.

Errors from Bridge node (between 10 and 100 before service failure).

ERROR core core/fetcher.go:191 fetcher: error receiving new height {"err": "rpc error: code = Unimplemented desc = unknown service tendermint.rpc.grpc.BlockAPI"} ERROR core core/fetcher.go:191 fetcher: error receiving new height {"err": "rpc error: code = Unimplemented desc = unknown service tendermint.rpc.grpc.BlockAPI"} ERROR core core/fetcher.go:191 fetcher: error receiving new height {"err": "rpc error: code = Unimplemented desc = unknown service tendermint.rpc.grpc.BlockAPI"}

until

Error: node: failed to start: fetching block by hash B93BBE20A0FBFDF955811B6420F8433904664D45DB4BF51022BE4200C1A1680D: rpc error: code = Unimplemented desc = unknown service tendermint.rpc.grpc.BlockAPI

Troubleshooting steps taken (and why this is opened under celestia-app):

Config parameters set according to docs (pruning = "none", kv, keep block history, etc).

grpcurl shows that the genisis block is retrievable from the core to the bridge, also tested from block 2:

bridge@bridge:grpcurl -plaintext -d '{ "height": "2" }' <archiveIP:grpcport> cosmos.base.tendermint.v1beta1.Service/GetBlockByHeight { "blockId": { "hash": "eNRdRVLZR/YTvZKDQ5RJ5nk01cj0fIlcHJM3xPRPkAY=", ......

but fails when using the tendermint lookup

bridge@bridge$ grpcurl -plaintext <archiveIP:grpcport> describe tendermint.rpc.grpc.BlockAPI Failed to resolve symbol "tendermint.rpc.grpc.BlockAPI": Symbol not found: tendermint.rpc.grpc.BlockAPI

Ran tcpdump listening on the core node and it saw the http2 tcp session being setup, but then being closed by the Core node when the bridge service started up.

This may also be a red herring, but it seems this merge celestiaorg/celestia-core@18a7372 wasn't included in the v3.3.0 release.

Please let me know if you need any additional info.

Version

Bridge: v0.21.5-mocha
celestia-appd: v3.3.0-mocha/v3.3.1-mocha

Steps to Reproduce

Run core node in archive mode on v3.3.1-mocha, init new bridge node on v0.21.5-mocha in archive mode. Using updated gRPC connectivity strings, attempt to connect to synched core node.