Zanod crashed on Raspberry PI 4 using lmbd

[apophis1329]

Hi,

i compiled zanod from sources following the instructions (using openssl-1.1.1t) - compile worked without any issues !
OS used was Raspberry PI OS:
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian

running the freshly compiled bits like this:
./zanod --no-predownload

sync starts and runs for some time before a crash occurs:
2024-Jan-16 00:11:01.135862 [P2P2][95.217.46.49:11121 OUT]>>>>>>>>> sync progress: 200 blocks added, now have 571121 of 2460797 ( 23.21% ) and 1889676 blocks left
2024-Jan-16 00:11:08.904551 [P2P9][95.217.46.49:11121 OUT]>>>>>>>>> sync progress: 199 blocks added, now have 571320 of 2460797 ( 23.22% ) and 1889477 blocks left
2024-Jan-16 00:11:17.055905 [P2P7][95.217.46.49:11121 OUT]>>>>>>>>> sync progress: 200 blocks added, now have 571520 of 2460797 ( 23.22% ) and 1889277 blocks left
2024-Jan-16 00:11:23.271447 [P2P2][95.217.46.49:11121 OUT]Recalculating median fee...
2024-Jan-16 00:11:23.607672 [P2P2][95.217.46.49:11121 OUT]Median fee recalculated for h = 571680 as 0.010000000000
2024-Jan-16 00:11:26.119902 [P2P2][95.217.46.49:11121 OUT]>>>>>>>>> sync progress: 200 blocks added, now have 571720 of 2460799 ( 23.23% ) and 1889079 blocks left
2024-Jan-16 00:11:30.100240 [P2P7][95.217.46.49:11121 OUT][ERROR] Location:
[set] @ /COINS/ZANO/zano/src/common/db_backend_lmdb.cpp:336
STACK
./zanod(_ZN5tools13get_callstackB5cxx11Ev+0x14) [0x557c1bdf18]
./zanod(_ZN5tools2db15lmdb_db_backend3setEmPKcmS3_m+0x164) [0x557c45f094]
./zanod(+0x30fe24) [0x557c3bfe24]
./zanod(_ZN8currency18blockchain_storage31update_spent_tx_flags_for_inputERKN6crypto4hashEmb+0x1f8) [0x557c3722b8]
./zanod(_ZN8currency18blockchain_storage31update_spent_tx_flags_for_inputEmmb+0xc8) [0x557c37296c]
./zanod(+0x2d940c) [0x557c38940c]
./zanod(_ZN8currency18blockchain_storage26add_transaction_from_blockERKNS_11transactionERKN6crypto4hashES7_mm+0x24c) [0x557c3898ec]
./zanod(_ZN8currency18blockchain_storage26handle_block_to_main_chainERKNS_5blockERKN6crypto4hashERNS_26block_verification_contextE+0xce0) [0x557c3a2ca0]
./zanod(_ZN8currency18blockchain_storage13add_new_blockERKNS_5blockERNS_26block_verification_contextE+0x6b4) [0x557c3a6b04]
./zanod(_ZN8currency4core13add_new_blockERKNS_5blockERNS_26block_verification_contextE+0x50) [0x557c3db454]
./zanod(_ZN8currency4core21handle_incoming_blockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_26block_verification_contextEb+0x1cc) [0x557c3dd99c]
./zanod(_ZN8currency27t_currency_protocol_handlerINS_4coreEE27handle_response_get_objectsEiRNS_27NOTIFY_RESPONSE_GET_OBJECTS7requestERNS_27currency_connection_contextE+0x145c) [0x557c2e3dbc]
./zanod(+0x3f8a58) [0x557c4a8a58]
./zanod(+0x3f1684) [0x557c4a1684]
./zanod(+0x1d8cf4) [0x557c288cf4]
./zanod(_ZN4epee9net_utils10connectionINS_5levin22async_protocol_handlerIN8nodetool24p2p_connection_context_tIN8currency27currency_connection_contextEEEEEE11handle_readERKN5boost6system10error_codeEm+
0xa8) [0x557c278668]
./zanod(_ZN5boost4asio6detail18completion_handlerINS1_17rewrapped_handlerINS1_7binder2INS1_15wrapped_handlerINS0_10io_context6strandENS_3_bi6bind_tIvNS_4_mfi3mf2IvN4epee9net_utils10connectionINSC_5lev
in22async_protocol_handlerIN8nodetool24p2p_connection_context_tIN8currency27currency_connection_contextEEEEEEERKNS_6system10error_codeEmEENS8_5list3INS8_5valueINS_10shared_ptrISN_EEEEPFNS_3argILi1EEEv
EPFNSY_ILi2EEEvEEEEENS1_26is_continuation_if_runningEEESP_mEES16_EEE11do_completeEPvPNS1_19scheduler_operationESR_m+0x124) [0x557c2abcb4]
./zanod(_ZN5boost4asio6detail23reactive_socket_recv_opINS0_17mutable_buffers_1ENS1_15wrapped_handlerINS0_10io_context6strandENS_3_bi6bind_tIvNS_4_mfi3mf2IvN4epee9net_utils10connectionINSB_5levin22asyn
c_protocol_handlerIN8nodetool24p2p_connection_context_tIN8currency27currency_connection_contextEEEEEEERKNS_6system10error_codeEmEENS7_5list3INS7_5valueINS_10shared_ptrISM_EEEEPFNS_3argILi1EEEvEPFNSX_I
Li2EEEvEEEEENS1_26is_continuation_if_runningEEENS1_18io_object_executorINS0_8executorEEEE11do_completeEPvPNS1_19scheduler_operationESQ_m+0x618) [0x557c29e4a8]
./zanod(+0x45ec7c) [0x557c50ec7c]
./zanod(_ZN4epee9net_utils18boosted_tcp_serverINS_5levin22async_protocol_handlerIN8nodetool24p2p_connection_context_tIN8currency27currency_connection_contextEEEEEE13worker_threadEv+0x2dc) [0x557c257fa
0]

zanod.log_crashing.gz

[cryptozoidberg]

@apophis1329 thanks for reporting the issue, we'll work on it as soon as we finish Zarcanum transition

[Derloda]

The issue is caused by the outdated liblmdb version used here which do not support 16K page: https://bugs.openldap.org/show_bug.cgi?id=9806

It is already fixed upstream, syncing the repository is most likely the only thing that needs to be done. It would also enable a native apple silicon build, which would greatly improves performance.

Edit: Unfortunately it also probably means the database couldn't be universally shared between 16K/4K page systems. In that case, enforcing --no-predownload (or providing a compatible database) would be mandatory on 16K page systems.

[cryptozoidberg]

@Derloda thanks for investigation of the issue, it's likely indeed time to upstream lmdb library, just need to make sure the new version would work smooth on all platform and won't behave different from old one in terms of blocks/txs processing (to avoid potential network splits). We'll plan this task for one of the nearest sprints.

[apophis1329]

Please let me know if you have a version ready for testing - very happy to help with testing on my different Raspis :-)

[Derloda]

Any update regarding the issue? It would be great to be able to compile natively for apple silicon.

[cryptozoidberg]

We had few conversations regarding support of Raspberry PI, and honestly i still not sure if it's good idea, mostly because of hardware limitations, especially with Zarcanum's transactions which is more heavier from computational perspective. Of course i'm not controlling this, and anyone can make Zano Raspberry-compatible, i just don't feel good about having a lot of this type of devices as Zano nodes.
Regarding apple silicon - is there any problems with this? We have codebase compiled for mobile app in native library for M-processors with no issues (https://github.com/hyle-team/zano_native_lib)

[Derloda]

For Apple Silicon, yes it's the same issue (lmdb in Zanod). It can't even synchronize correctly because of that, wallet usage is not the issue.

[sowle]

I took a look at the issue.
Indeed, the issue 9806 was fixed in liblmdb, effective starting with version 0.9.30 by this commit.

However, I can't see how I can manually set page size if I'd like to convert a DB with one page size into another. Key threshold size is always defiled as:

			/* Threshold number of keys considered "small" */
			keythresh = env->me_psize >> 7;

(https://github.com/LMDB/lmdb/blob/LMDB_0.9.31/libraries/liblmdb/mdb.c#L8766)

Where me_psize is derived from the system page size and cannot be changed, if I understand it correctly.

UPD. Note: ver. 0.9.31 is the latest on github mirror, but isn't the latest in openldap git, where 0.9.33 exists: https://git.openldap.org/openldap/openldap/-/tags/LMDB_0.9.33 However, it's still unclear is it possible to create a 16k page DB on 4k page system.