wasm queries and state sync broken after upgrading to v0.51.0

[dasanchez]

Hi!

We upgraded the Cosmos Hub ICS testnet from Gaia v19.0.0-rc0 to Gaia v19.0.0-rc3 through a coordinated upgrade, and we are seeing issues now:

• State sync nodes are not able to create snapshots

Aug 02 09:27:44 provider-state-sync-02 cosmovisor[2385]: 9:27AM INF creating state snapshot height=7648000 module=server
Aug 02 09:35:58 provider-state-sync-02 cosmovisor[2385]: 9:35AM ERR failed to create state snapshot err="failed to generate snapshot chunk 7: Error calling the VM: Cache error: Error opening Wasm file for reading" height=7648000 module=server

• We can no longer query a contract's state

gaiad q wasm contract-state smart cosmos14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9s4hmalr '{"get_count":{}}'
Error: rpc error: code = Unknown desc = Error calling the VM: Cache error: Error opening Wasm file for reading: wasmvm error: unknown request

• We can no longer execute contracts

gaiad tx wasm execute cosmos14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9s4hmalr '{"increment":{}}' --from account --gas auto --gas-adjustment 1.5 --gas-prices 0.005uatom
Error: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: Error calling the VM: Cache error: Error opening Wasm file for reading: wasmvm error [CosmWasm/[email protected]/x/wasm/keeper/keeper.go:415] with gas used: '118525': unknown request

go.mod dependencies:

• Gaia v19.0.0-rc0 (pre-upgrade):

  • wasmd v0.50.0
  • wasmvm v1.5.0

• Gaia v19.0.0-rc3 (post-upgrade):

  • wasmd v0.51.0
  • wasmvm/v2 v2.0.0

Do you know how we can fix these? Thanks!

[pinosu]

Hey @dasanchez, did you follow the migration guide when migrating from wasmvm v1.5.0 to v2.0.0?

[chipshort]

This looks like you are missing the wasm files or they have incorrect permissions. They should be in the home directory somewhere. I don't remember the exact path.

[webmaster128]

Can you check if there is anything in ~/.gaiad/wasm/wasm/state/wasm? This is where the Wasm blobs of the node are stored. When you do a snapshot you should add ~/.gaiad/wasm and remove ~/.gaiad/wasm/wasm/cache.

[dasanchez]

Thanks @pinosu @chipshort @webmaster128 !
The fix was indeed related to the .gaia/wasm folders: after upgrading to an earlier rc of Gaia v19, the expected location of the wasm blobs changed.
After merging the contents of .gaia/wasm/cache and .gaia/wasm/state with .gaia/wasm/wasm/cache and .gaia/wasm/wasm/state the issue went away and we were able to state sync and query contracts again.
You can close this issue, I appreciate you looking into this!

[webmaster128]

Okay cool.

This wasm/wasm structure is more or less a bug we carry around for a long time and it was never important enough to fix it. Almost every CosmWasm chain has this duplicated folder structure. Just Juno managed to change it ti simple wasm. So it might be related to Reece doing the original integration to the Hub codebase.