Skip to content

fullnode crush #1305

Description

@zavndw

docker container full-node stops after error

in logs last:

2025-09-19T16:04:50.527415Z  INFO maptos_opt_executor::background::transaction_pipe: TX executed successfully. tx_hash=98e4d31d sender=0x9ff88f66128fa15c6b46a7177814e48b280c9fe62ef1d73d05c2f3deb307456f sequence_number=470
2025-09-19T16:04:50.527430Z  INFO maptos_opt_executor::background::transaction_pipe: TX executed successfully. tx_hash=966f6bb2 sender=0xa76011908551ef38768cdc9cc576346514fbbbfb2260aff42768467e68960ca1 sequence_number=1031
2025-09-19T16:04:50.527463Z  INFO movement_timing: decrementing_transactions_in_flight count=2 current=0
2025-09-19T16:04:50.527523Z  INFO process_block_from_da{block_id=8ecb3aa9260686c5f7b8387ebbaa83a19a03e7813b972be0e71f79700ebec7b8}: movement_full_node::node::tasks::execute_settle: Skipping settlement block_id=[142, 203, 58, 169, 38, 6, 134, 197, 247, 184, 56, 126, 187, 170, 131, 161, 154, 3, 231, 129, 59, 151, 43, 224, 231, 31, 121, 112, 14, 190, 199, 184]
2025-09-19T16:04:50.527537Z  INFO movement_full_node::node::tasks::execute_settle: New state after execution: Some(ExecutionState { block_height: 7117640, ledger_timestamp: 1758297890466781, ledger_version: 27204552 })
2025-09-19T16:04:50.539729Z  INFO aptos_indexer_grpc_utils::counters: [Indexer Table Info] Processed batch successfully start_version=27204549 end_version=27204553 num_transactions=4 duration_in_secs=0.003414298 service_type="table_info_service" step="1"
2025-09-19T16:04:50.540098Z  INFO aptos_indexer_grpc_utils::counters: [Indexer Table Info] Processed successfully start_version=27204549 end_version=27204552 num_transactions=4 duration_in_secs=1.9706803339999999 service_type="table_info_service" step="2"
2025-09-19T16:05:13.296555Z ERROR movement_full_node::node::tasks::execute_settle: Da client stream channel timeout because it's idle. Exit

When save logs, I received a trace from the standout:

thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.41.1/src/runtime/blocking/shutdown.rs:51:21:
Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: tokio::runtime::blocking::shutdown::Receiver::wait
   3: tokio::runtime::blocking::pool::BlockingPool::shutdown
   4: core::ptr::drop_in_place<tokio::runtime::runtime::Runtime>
   5: core::ptr::drop_in_place<<maptos_dof_execution::v1::Executor as maptos_dof_execution::DynOptFinExecutor>::background::{{closure}}>
   6: tracing::instrument::_::<impl core::ops::drop::Drop for tracing::instrument::Instrumented<T>>::drop
   7: core::ptr::drop_in_place<tokio::runtime::task::core::Stage<tracing::instrument::Instrumented<<maptos_dof_execution::v1::Executor as maptos_dof_execution::DynOptFinExecutor>::background::{{closure}}>>>
   8: tokio::runtime::task::harness::Harness<T,S>::shutdown
   9: tokio::runtime::task::list::OwnedTasks<S>::close_and_shutdown_all
  10: tokio::runtime::scheduler::multi_thread::worker::Context::run
  11: tokio::runtime::context::scoped::Scoped<T>::set
  12: tokio::runtime::context::runtime::enter_runtime
  13: tokio::runtime::scheduler::multi_thread::worker::run
  14: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
  15: tokio::runtime::task::core::Core<T,S>::poll
  16: tokio::runtime::task::harness::Harness<T,S>::poll
  17: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Error: Block execution loop break. Node need to be restarted.

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
   1: movement_full_node::node::tasks::execute_settle::Task<E,S>::run::{{closure}}
   2: <core::pin::Pin<P> as core::future::future::Future>::poll
   3: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
   4: tokio::runtime::task::core::Core<T,S>::poll
   5: tokio::runtime::task::harness::Harness<T,S>::poll
   6: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
   7: tokio::runtime::scheduler::multi_thread::worker::Context::run
   8: tokio::runtime::context::scoped::Scoped<T>::set
   9: tokio::runtime::context::runtime::enter_runtime
  10: tokio::runtime::scheduler::multi_thread::worker::run
  11: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
  12: tokio::runtime::task::core::Core<T,S>::poll
  13: tokio::runtime::task::harness::Harness<T,S>::poll
  14: tokio::runtime::blocking::pool::Inner::run
  15: std::sys::backtrace::__rust_begin_short_backtrace
  16: core::ops::function::FnOnce::call_once{{vtable.shim}}
  17: std::sys::pal::unix::thread::Thread::new::thread_start
  18: start_thread
  19: __clone

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions