Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cd): deploy instances with attached cached states #8868

Merged
merged 5 commits into from
Sep 19, 2024

Conversation

gustavovalverde
Copy link
Member

@gustavovalverde gustavovalverde commented Sep 17, 2024

Motivation

We need to have long-live instances deployed, but we need to deploy instances beforehand with the cached state attached when merging PRs into the main branch, to test instances are syncing and running correctly.

Fixes: #6894

Specifications & References

This is based in the refactor made in:

Solution

  • Deploy CD instances when pushing to the main branch
  • Get available cached states, with a preference on the main branch cached states, using sub-find-cached-disks.yml
  • Use the "${DISK_PARAMS}" as we do in CI
  • Reorganize gcloud parameters into a more logical way

Tests

Logs from manually deployed instance with cached state
Thank you for running a mainnet zebrad 1.9.0 node!
You're helping to strengthen the network and contributing to a social good :)
2024-09-17T17:21:15.691120Z  INFO zebrad::components::tracing::component: started tracing component filter="info" TRACING_STATIC_MAX_LEVEL=LevelFilter::INFO LOG_STATIC_MAX_LEVEL=Info
2024-09-17T17:21:15.691156Z  INFO zebrad::components::tracing::component: installed sentry tracing layer
2024-09-17T17:21:15.691159Z  INFO zebrad::components::tracing::component: set 'tracing.progress_bar ="summary"' in zebrad.toml to activate progress bars
2024-09-17T17:21:15.691161Z  INFO zebrad::application: Diagnostic Metadata:
version: 1.9.0
Zcash network: Mainnet
running state version: 26.0.0
initial disk state version: 26.0.0
features: default,default_release_binaries,getblocktemplate_rpcs,howudoin,indicatif,progress_bar,release_max_level_info,sentry
git commit: 7fe1458
target triple: x86_64-unknown-linux-gnu
rust compiler: 1.79.0
rust release date: 2024-06-10
optimization level: 3
debug checks: false
2024-09-17T17:21:15.691163Z  INFO zebrad::application: loaded zebrad config config_path=Some("/etc/zebrad/zebrad.toml") config=ZebradConfig { consensus: Config { checkpoint_sync: true }, metrics: Config { endpoint_addr: None }, network: Config { listen_addr: 0.0.0.0:8233, external_addr: None, network: Mainnet, initial_mainnet_peers: {"dnsseed.z.cash:8233", "dnsseed.str4d.xyz:8233", "mainnet.seeder.zfnd.org:8233", "mainnet.is.yolo.money:8233"}, initial_testnet_peers: {"dnsseed.testnet.z.cash:18233", "testnet.seeder.zfnd.org:18233", "testnet.is.yolo.money:18233"}, cache_dir: IsEnabled(true), peerset_initial_target_size: 25, crawl_new_peer_interval: 61s, max_connections_per_ip: 1 }, state: Config { cache_dir: "/var/cache/zebrad-cache", ephemeral: false, delete_old_database: true, debug_stop_at_height: None, debug_validity_check_interval: None }, tracing: Config { inner: InnerConfig { use_color: false, force_use_color: false, filter: None, buffer_limit: 128000, endpoint_addr: None, flamegraph: None, progress_bar: None, log_file: None, use_journald: false } }, sync: Config { download_concurrency_limit: 50, checkpoint_verify_concurrency_limit: 1000, full_verify_concurrency_limit: 20, parallel_cpu_threads: 0 }, mempool: Config { tx_cost_limit: 80000000, eviction_memory_time: 3600s, debug_enable_at_height: None }, rpc: Config { listen_addr: Some(0.0.0.0:8232), indexer_listen_addr: None, parallel_cpu_threads: 0, debug_force_finished_sync: false }, mining: Config { miner_address: None, extra_coinbase_data: None, debug_like_zcashd: true } }
2024-09-17T17:21:15.691270Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::application: initialized rayon thread pool for CPU-bound tasks num_threads=4
2024-09-17T17:21:15.691711Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: Starting zebrad
2024-09-17T17:21:15.691782Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: initializing node state
2024-09-17T17:21:15.697274Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: opening database, this may take a few minutes
2024-09-17T17:21:15.697472Z  WARN zebra_state::service::finalized_state::disk_db: could not canonicalize "/var/cache/zebrad-cache/state/v25/mainnet": No such file or directory (os error 2)
2024-09-17T17:21:15.697509Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: trying to open current database format running_version=26.0.0
2024-09-17T17:21:15.697754Z  INFO zebra_state::service::finalized_state::disk_db: the open file limit is high enough for Zebra current_limit=1048576 min_limit=512 ideal_limit=1024
2024-09-17T17:22:33.314400Z  INFO zebra_state::service::finalized_state::disk_db: Opened Zebra state cache at /var/cache/zebrad-cache/state/v26/mainnet
2024-09-17T17:22:33.316785Z  INFO zebra_state::service::finalized_state: loaded Zebra state cache tip=Some((Height(2647533), block::Hash("00000000005801784e8d162f7a4894e19ceb6f605d2887ed272a436d5cb73c69")))
2024-09-17T17:22:33.316810Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: checking database format produced by a previous zebra instance is current and valid running_version=26.0.0
2024-09-17T17:22:33.318406Z  INFO zebra_state::service: starting legacy chain check
2024-09-17T17:22:33.319292Z  INFO zebra_state::service: cached state consensus branch is valid: no legacy chain found
2024-09-17T17:22:33.319341Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: logging database metrics on startup
2024-09-17T17:22:33.319393Z  INFO {zebrad="7fe1458" net="Main"}: zebra_state::service::finalized_state::disk_db: the open file limit is high enough for Zebra current_limit=1048576 min_limit=512 ideal_limit=1024
2024-09-17T17:22:33.334838Z  INFO {zebrad="7fe1458" net="Main"}: zebra_state::service::finalized_state::disk_db: Total Database Disk Size: 532.9 GB
2024-09-17T17:22:33.334866Z  INFO {zebrad="7fe1458" net="Main"}: zebra_state::service::finalized_state::disk_db: Total Live Data Disk Size: 265.9 GB
2024-09-17T17:22:33.334869Z  INFO {zebrad="7fe1458" net="Main"}: zebra_state::service::finalized_state::disk_db: Total Database Memory Size: 49.2 KB
2024-09-17T17:22:33.334921Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: initializing network
2024-09-17T17:22:33.334967Z  INFO {zebrad="7fe1458" net="Main"}:open_listener{addr=0.0.0.0:8233}: zebra_network::peer_set::initialize: Trying to open Zcash protocol endpoint at 0.0.0.0:8233...
2024-09-17T17:22:33.335057Z  INFO {zebrad="7fe1458" net="Main"}:open_listener{addr=0.0.0.0:8233}: zebra_network::peer_set::initialize: Opened Zcash protocol endpoint at 0.0.0.0:8233
2024-09-17T17:22:33.335485Z  INFO {zebrad="7fe1458" net="Main"}: zebra_network::address_book_updater: starting the address book updater
2024-09-17T17:22:33.386044Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::config: resolved seed peer IP addresses seed="dnsseed.str4d.xyz:8233" remote_ip_count=28
2024-09-17T17:22:33.418567Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::config: resolved seed peer IP addresses seed="mainnet.seeder.zfnd.org:8233" remote_ip_count=25
2024-09-17T17:22:33.468729Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::config: DNS error resolving peer IP addresses host="dnsseed.z.cash:8233" e=Custom { kind: Uncategorized, error: "failed to lookup address information: No address associated with hostname" }
2024-09-17T17:22:33.468760Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::config: Seed peer DNS resolution failed, checking for addresses from other seed peers host="dnsseed.z.cash:8233" attempts=1
2024-09-17T17:22:37.492217Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::config: DNS error resolving peer IP addresses host="mainnet.is.yolo.money:8233" e=Custom { kind: Uncategorized, error: "failed to lookup address information: Temporary failure in name resolution" }
2024-09-17T17:22:37.492264Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::config: Seed peer DNS resolution failed, checking for addresses from other seed peers host="mainnet.is.yolo.money:8233" attempts=1
2024-09-17T17:22:37.492444Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: limiting the initial peers list from 47 to 25
2024-09-17T17:22:37.492507Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: connecting to initial peer set ipv4_peer_count=24 ipv6_peer_count=1
2024-09-17T17:22:38.614578Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: an initial peer connection failed successes=8 errors=1 addr=v4redacted:8233 e=Connection refused (os error 111)
2024-09-17T17:22:38.910534Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: an initial peer connection failed successes=9 errors=2 addr=v4redacted:8233 e=Serialization error
2024-09-17T17:22:39.558242Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: an initial peer connection failed successes=14 errors=3 addr=v4redacted:8233 e=Connection refused (os error 111)
2024-09-17T17:22:39.893822Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: an initial peer connection failed successes=17 errors=4 addr=v6redacted:8233 e=Network is unreachable (os error 101)
2024-09-17T17:22:40.594790Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: an initial peer connection failed successes=18 errors=5 addr=v4redacted:8233 e=request timed out
2024-09-17T17:22:41.994574Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: an initial peer connection failed successes=18 errors=6 addr=v4redacted:8233 e=request timed out
2024-09-17T17:22:42.295222Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: an initial peer connection failed successes=18 errors=7 addr=v4redacted:8233 e=request timed out
2024-09-17T17:22:42.295264Z  INFO {zebrad="7fe1458" net="Main"}:add_initial_peers: zebra_network::peer_set::initialize: finished connecting to initial seed and disk cache peers handshake_success_total=18 handshake_error_total=7 outbound_connections=19
2024-09-17T17:22:42.295317Z  INFO {zebrad="7fe1458" net="Main"}: zebra_network::peer_set::initialize: sending initial request for peers active_initial_peer_count=19
2024-09-17T17:22:42.427770Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: initializing verifiers
2024-09-17T17:22:42.427819Z  INFO {zebrad="7fe1458" net="Main"}:crawl_and_dial{new_peer_interval=61s}: zebra_network::peer_set::initialize: starting the peer address crawler crawl_new_peer_interval=61s outbound_connections=18
2024-09-17T17:22:42.427897Z  INFO {zebrad="7fe1458" net="Main"}:init{config=Config { checkpoint_sync: true } network=Mainnet}: zebra_consensus::router: starting state checkpoint validation
2024-09-17T17:22:42.436109Z  INFO {zebrad="7fe1458" net="Main"}:init{config=Config { checkpoint_sync: true } network=Mainnet}: zebra_consensus::router: initializing block verifier router tip=Some((Height(2647533), block::Hash("00000000005801784e8d162f7a4894e19ceb6f605d2887ed272a436d5cb73c69"))) max_checkpoint_height=Height(2625206)
2024-09-17T17:22:42.436160Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: initializing syncer
2024-09-17T17:22:42.436172Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: initializing mempool
2024-09-17T17:22:42.436178Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: fully initializing inbound peer request handler
2024-09-17T17:22:42.436189Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: spawning RPC server
2024-09-17T17:22:42.436194Z  INFO {zebrad="7fe1458" net="Main"}: zebra_rpc::server: Trying to open RPC endpoint at 0.0.0.0:8232...
2024-09-17T17:22:42.436949Z  INFO {zebrad="7fe1458" net="Main"}: zebra_rpc::server: Opened RPC endpoint at 0.0.0.0:8232
2024-09-17T17:22:42.437084Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: spawning block gossip task
2024-09-17T17:22:42.437156Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: spawning mempool queue checker task
2024-09-17T17:22:42.437163Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: spawning mempool transaction gossip task
2024-09-17T17:22:42.437168Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: spawning delete old databases task
2024-09-17T17:22:42.437172Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::gossip: initializing block gossip task
2024-09-17T17:22:42.437175Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: spawning progress logging task
2024-09-17T17:22:42.437179Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: spawning end of support checking task
2024-09-17T17:22:42.437187Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: spawning mempool crawler task
2024-09-17T17:22:42.437191Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: spawning syncer task
2024-09-17T17:22:42.437197Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::commands::start: spawned initial Zebra tasks
2024-09-17T17:22:42.437210Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::queue_checker: initializing mempool queue checker task
2024-09-17T17:22:42.437304Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::end_of_support: Starting end of support task
2024-09-17T17:22:42.437312Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::crawler: initializing mempool crawler task
2024-09-17T17:22:42.437409Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2647533))
2024-09-17T17:22:42.437450Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: initializing transaction gossip task
2024-09-17T17:22:42.437505Z  INFO {zebrad="7fe1458" net="Main"}: zebra_state::config: checking for old database versions db_kind="state"
2024-09-17T17:22:42.438135Z  INFO {zebrad="7fe1458" net="Main"}: zebra_state::config: finished old database version cleanup task
2024-09-17T17:22:42.884623Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync:obtain_tips: zebra_consensus::primitives::groth16::params: checking and loading Zcash Sapling and Sprout parameters
2024-09-17T17:22:43.176591Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync:obtain_tips: zebra_consensus::primitives::groth16::params: Zcash Sapling and Sprout parameters loaded and verified
2024-09-17T17:22:47.081963Z  INFO {zebrad="7fe1458" net="Main"}:init{config=Config { checkpoint_sync: true } network=Mainnet}: zebra_consensus::router: finished state checkpoint validation
2024-09-17T17:22:52.438387Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::end_of_support: Checking if Zebra release is inside support range ...
2024-09-17T17:22:52.438462Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::end_of_support: Zebra release is supported until block 2707140, please report bugs at https://github.com/ZcashFoundation/zebra/issues
2024-09-17T17:22:52.499965Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: database format is valid running_version=26.0.0 inital_disk_version=26.0.0
2024-09-17T17:23:02.429589Z  INFO {zebrad="7fe1458" net="Main"}:peer_cache_updater: zebra_network::config: updated cached peer IP addresses cached_ip_count=25 peer_cache_file="/root/.cache/zebra/network/mainnet.peers"
2024-09-17T17:23:33.615117Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync:try_to_sync_once{extra_hashes={}}: zebrad::components::sync: extending tips tips.len=1 in_flight=19 extra_hashes=0 lookahead_limit=20 state_tip=Some(Height(2648013))
2024-09-17T17:23:42.457715Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: estimated progress to chain tip sync_percent=99.932% current_height=Height(2648085) network_upgrade=Nu5 remaining_sync_blocks=1804 time_since_last_state_block=0s
2024-09-17T17:24:27.977414Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync:try_to_sync_once{extra_hashes={}}: zebrad::components::sync: extending tips tips.len=1 in_flight=19 extra_hashes=0 lookahead_limit=20 state_tip=Some(Height(2648511))
2024-09-17T17:24:42.471445Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: estimated progress to chain tip sync_percent=99.955% current_height=Height(2648702) network_upgrade=Nu5 remaining_sync_blocks=1185 time_since_last_state_block=0s
2024-09-17T17:25:52.263769Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: estimated progress to chain tip sync_percent=99.962% current_height=Height(2648870) network_upgrade=Nu5 remaining_sync_blocks=1017 time_since_last_state_block=0s
2024-09-17T17:26:17.534260Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync:try_to_sync_once{extra_hashes={}}: zebrad::components::sync: extending tips tips.len=1 in_flight=19 extra_hashes=0 lookahead_limit=20 state_tip=Some(Height(2649009))
2024-09-17T17:26:42.228394Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync:try_to_sync_once{extra_hashes={}}: zebrad::components::sync: extending tips tips.len=1 in_flight=19 extra_hashes=0 lookahead_limit=20 state_tip=Some(Height(2649507))
2024-09-17T17:26:52.276496Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: estimated progress to chain tip sync_percent=99.994% current_height=Height(2649709) network_upgrade=Nu5 remaining_sync_blocks=171 time_since_last_state_block=0s
2024-09-17T17:26:59.675275Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync:try_to_sync_once{extra_hashes={}}: zebrad::components::sync: extending tips tips.len=1 in_flight=19 extra_hashes=0 lookahead_limit=20 state_tip=Some(Height(2649861))
2024-09-17T17:26:59.768889Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:26:59.768978Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649863))
2024-09-17T17:27:52.292675Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: estimated progress to chain tip sync_percent=100.000% current_height=Height(2649880) network_upgrade=Nu5 remaining_sync_blocks=3 time_since_last_state_block=0s
2024-09-17T17:28:02.431012Z  INFO {zebrad="7fe1458" net="Main"}:peer_cache_updater: zebra_network::config: updated cached peer IP addresses cached_ip_count=31 peer_cache_file="/root/.cache/zebra/network/mainnet.peers"
2024-09-17T17:28:06.769536Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649880))
2024-09-17T17:28:06.998158Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:28:06.998187Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649880))
2024-09-17T17:28:52.305057Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: estimated progress to chain tip sync_percent=100.000% current_height=Height(2649880) network_upgrade=Nu5 remaining_sync_blocks=4 time_since_last_state_block=1m
2024-09-17T17:29:13.998930Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649880))
2024-09-17T17:29:14.516596Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync:try_to_sync_once{extra_hashes={}}: zebrad::components::sync: extending tips tips.len=1 in_flight=2 extra_hashes=0 lookahead_limit=20 state_tip=Some(Height(2649880))
2024-09-17T17:29:14.562472Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:29:14.562498Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649880))
2024-09-17T17:29:15.110599Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=00000000011dbb5ed56bc9599dc98876151d9d4b690d99067ee88ba98ea97f23}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649882)
2024-09-17T17:29:15.154498Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000001e2fcd96976e10857712ebf286f488a9a1333e9df06368e628f23d0}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649883)
2024-09-17T17:29:15.209883Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=000000000025cf255bc4c714929fe57c3b6c6d2c03b65627efe572aecac2ea5d}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649884)
2024-09-17T17:29:15.460940Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}: zebrad::components::mempool: activating mempool: Zebra is close to the tip tip_height=Height(2649884)
2024-09-17T17:29:16.259749Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:29:23.261347Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:29:45.369905Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=00000000004f4a97f98f5dcbacb67b82fec236e2c9054bbbed0d8c27acbc663d}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649885)
2024-09-17T17:29:52.318825Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649885) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:30:07.832430Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:30:07.957176Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=000000000092534942f391fb6c22c312986e48ac4ed935a93b28285ce98d78f4}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649886)
2024-09-17T17:30:21.563870Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649886))
2024-09-17T17:30:23.845188Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000001e90e9bb51a00149125cfb88e1578c06706e09d518ee61b90c61573}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649887)
2024-09-17T17:30:27.564905Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:30:27.564933Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649887))
2024-09-17T17:30:52.333839Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649887) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:31:34.566010Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649887))
2024-09-17T17:31:40.566637Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:31:40.566671Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649887))
2024-09-17T17:31:52.346397Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649887) network_upgrade=Nu5 remaining_sync_blocks=1 time_since_last_state_block=1m
2024-09-17T17:32:00.740296Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=000000000013e2a87e41ce325d11d7411e0671ed0ac7688f368aeea8e3f1019c}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649888)
2024-09-17T17:32:06.267349Z  INFO {zebrad="7fe1458" net="Main"}:crawl_and_dial{new_peer_interval=61s}:crawl{should_always_dial=false}: zebra_network::peer_set::candidate_set: timeout waiting for peer service readiness or peer responses
2024-09-17T17:32:47.568015Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649888))
2024-09-17T17:32:52.358993Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649888) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:32:53.569830Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:32:53.569858Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649888))
2024-09-17T17:33:02.431974Z  INFO {zebrad="7fe1458" net="Main"}:peer_cache_updater: zebra_network::config: updated cached peer IP addresses cached_ip_count=35 peer_cache_file="/root/.cache/zebra/network/mainnet.peers"
2024-09-17T17:33:38.963497Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=000000000041253fce1f98a2ea584751504eddf214bf91efa344e88090b7b696}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649889)
2024-09-17T17:33:43.530688Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=000000000104af7f6a60e6c022fb830aba1edb51d7250428d4f97996d17f14dc}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649890)
2024-09-17T17:33:52.374398Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649890) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:34:00.570907Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649890))
2024-09-17T17:34:06.573299Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:34:06.573398Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649890))
2024-09-17T17:34:21.433763Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:34:28.495719Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:34:52.388233Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649890) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=1m
2024-09-17T17:35:06.983117Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:35:13.574480Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649890))
2024-09-17T17:35:19.576001Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:35:19.576026Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649890))
2024-09-17T17:35:45.625269Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=00000000019c29315b5e17eecb636f8114064e82425a26bbe6fbbe10b7a1ce80}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649891)
2024-09-17T17:35:52.401073Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649891) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:36:26.577391Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649891))
2024-09-17T17:36:29.249773Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000001143b2f643872514320196a8574aac4848db73e53115a806c9a88ea}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649892)
2024-09-17T17:36:29.509188Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:36:29.509235Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649892))
2024-09-17T17:36:42.299393Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:36:52.415388Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649892) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:37:16.484088Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:37:36.509838Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649892))
2024-09-17T17:37:42.512015Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:37:42.512038Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649892))
2024-09-17T17:37:52.428493Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649892) network_upgrade=Nu5 remaining_sync_blocks=1 time_since_last_state_block=1m
2024-09-17T17:38:02.433791Z  INFO {zebrad="7fe1458" net="Main"}:peer_cache_updater: zebra_network::config: updated cached peer IP addresses cached_ip_count=36 peer_cache_file="/root/.cache/zebra/network/mainnet.peers"
2024-09-17T17:38:28.896886Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000000a9e08307734f5f22ff0ae5c0c12eb1568c3eeeebf3bbf5f529f3f0}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649893)
2024-09-17T17:38:32.196571Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=000000000005557684f70e3f732d99a76ac26d84cb3ec2803deb4e94dea156e3}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649894)
2024-09-17T17:38:49.512685Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649894))
2024-09-17T17:38:52.444074Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649894) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:38:55.514511Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:38:55.514568Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649894))
2024-09-17T17:39:13.266254Z  INFO {zebrad="7fe1458" net="Main"}:crawl_and_dial{new_peer_interval=61s}:crawl{should_always_dial=false}: zebra_network::peer_set::candidate_set: timeout waiting for peer service readiness or peer responses
2024-09-17T17:39:52.455767Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649894) network_upgrade=Nu5 remaining_sync_blocks=1 time_since_last_state_block=1m
2024-09-17T17:40:02.515910Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649894))
2024-09-17T17:40:08.517173Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:40:08.517206Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649894))
2024-09-17T17:40:35.154262Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:40:52.469753Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649894) network_upgrade=Nu5 remaining_sync_blocks=1 time_since_last_state_block=2m
2024-09-17T17:40:52.812120Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000001b0f05fc8eececbe73ee82c6fb05d9f7382a8aa15a049a52c42082b}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649895)
2024-09-17T17:41:02.958886Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000000aca2a57abf0a1fac6754fecf90c5d8536cefdb88f3c17402fb7a17}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649896)
2024-09-17T17:41:15.266137Z  INFO {zebrad="7fe1458" net="Main"}:crawl_and_dial{new_peer_interval=61s}:crawl{should_always_dial=false}: zebra_network::peer_set::candidate_set: timeout waiting for peer service readiness or peer responses
2024-09-17T17:41:15.518012Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649896))
2024-09-17T17:41:21.519071Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:41:21.519117Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649896))
2024-09-17T17:41:51.461462Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:41:52.485654Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649896) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:41:52.586937Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000000dd81c24916dc643be9750a10e8336ef568e6f8264e986d487351f1}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649897)
2024-09-17T17:42:28.520183Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649897))
2024-09-17T17:42:34.521878Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:42:34.521921Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649897))
2024-09-17T17:42:52.500265Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649897) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:43:02.435642Z  INFO {zebrad="7fe1458" net="Main"}:peer_cache_updater: zebra_network::config: updated cached peer IP addresses cached_ip_count=39 peer_cache_file="/root/.cache/zebra/network/mainnet.peers"
2024-09-17T17:43:14.117582Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:43:15.968562Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=00000000015555a830d6624876d741e986d6d215d7134fd8cd668421104db952}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649898)
2024-09-17T17:43:21.118752Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:43:28.119917Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:43:33.129841Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=000000000093200273f3fb2cdc4bd07793c22aaa9cf6c1e41090aa680fb44615}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649899)
2024-09-17T17:43:34.052586Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=00000000006803d7ac08dc4f5345718d1ccd44fe48fb50eed1b633595af3c7bc}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649900)
2024-09-17T17:43:37.510699Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool: resetting mempool: switched best chain, skipped blocks, or activated network upgrade tip_height=Height(2649900)
2024-09-17T17:43:37.510734Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool: activating mempool: Zebra is close to the tip tip_height=Height(2649900)
2024-09-17T17:43:37.510738Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool: re-verifying mempool transactions after a chain fork transactions=3
2024-09-17T17:43:41.522894Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649900))
2024-09-17T17:43:44.397006Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:43:44.397030Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649900))
2024-09-17T17:43:44.520403Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=00000000016e56327ea3b8d043f67f5de608b24e536bb51d3d4b3a919a2d5f3f}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649901)
2024-09-17T17:43:49.444893Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=00000000012c9e190f40c09261fbc4c8a5cb2dbe475970a6ef41120cc18a3691}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649902)
2024-09-17T17:43:52.513715Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649902) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:44:21.765131Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000001ada719a7db9bf34a236f3ec2a08bd9a7a609428663023ecd57a8cd}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649903)
2024-09-17T17:44:51.397765Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649903))
2024-09-17T17:44:52.529317Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649903) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:44:56.450700Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000000b6eb7c6d4326cdbac7241faafee1da3a6e7b35039fefe5d541f237}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649904)
2024-09-17T17:44:56.594158Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:44:56.594179Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649904))
2024-09-17T17:45:03.575061Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000001258b33e2e6e0c0e540864119b973d555a5e16bb85e838f50dac484}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649905)
2024-09-17T17:45:52.544838Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649905) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:46:03.595017Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649905))
2024-09-17T17:46:09.597351Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:46:09.597377Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649905))
2024-09-17T17:46:52.558275Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649905) network_upgrade=Nu5 remaining_sync_blocks=1 time_since_last_state_block=1m
2024-09-17T17:47:04.935841Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:47:16.598858Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649905))
2024-09-17T17:47:22.601209Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:47:22.601272Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649905))
2024-09-17T17:47:27.131255Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0000000000a46288b6c5987a1c05ba2f7270f5b3080c75cbc492e32698666ff3}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649906)
2024-09-17T17:47:52.572979Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649906) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s
2024-09-17T17:48:02.437575Z  INFO {zebrad="7fe1458" net="Main"}:peer_cache_updater: zebra_network::config: updated cached peer IP addresses cached_ip_count=41 peer_cache_file="/root/.cache/zebra/network/mainnet.peers"
2024-09-17T17:48:20.099069Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=00000000012bc449bd702f23660e17c01de3959129f66211356dfcb725938ce5}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649907)
2024-09-17T17:48:29.601554Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649907))
2024-09-17T17:48:35.603226Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:48:35.603294Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649907))
2024-09-17T17:48:38.899566Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="mempool"}:mempool: zebrad::components::mempool: answered mempool request req=TransactionIds res_count=0
2024-09-17T17:48:52.585350Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649907) network_upgrade=Nu5 remaining_sync_blocks=1 time_since_last_state_block=0s
2024-09-17T17:49:08.404645Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:49:15.405953Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2024-09-17T17:49:42.603769Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: starting sync, obtaining new tips state_tip=Some(Height(2649907))
2024-09-17T17:49:48.604531Z  INFO {zebrad="7fe1458" net="Main"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2024-09-17T17:49:48.604581Z  INFO {zebrad="7fe1458" net="Main"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(2649907))
2024-09-17T17:49:52.599932Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649907) network_upgrade=Nu5 remaining_sync_blocks=1 time_since_last_state_block=1m
2024-09-17T17:50:12.571568Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=000000000057858e62a3cc13165eebf8431e7046dced2fc2278fbb3ca552f3bf}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649908)
2024-09-17T17:50:28.418155Z  INFO {zebrad="7fe1458" net="Main"}:{peer=Out("v4redacted:8233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=00000000009e1e98dcfe515e854def87c51ed0f5eb4924bf0f4853204b814cb0}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(2649909)
2024-09-17T17:50:52.612987Z  INFO {zebrad="7fe1458" net="Main"}: zebrad::components::sync::progress: finished initial sync to chain tip, using gossiped blocks sync_percent=100.000% current_height=Height(2649909) network_upgrade=Nu5 remaining_sync_blocks=0 time_since_last_state_block=0s

Follow-up Work

PR Author's Checklist

  • The PR name will make sense to users.
  • The PR provides a CHANGELOG summary.
  • The solution is tested.
  • The documentation is up to date.
  • The PR has a priority label.

PR Reviewer's Checklist

  • The PR Author's checklist is complete.
  • The PR resolves the issue.

We've been using multiple approaches to locate and retrieve cached states in GCP. However, this has made it difficult to reuse the same methods across new workflows or different scenarios.

To address this, we've streamlined the process to make it more reusable in other contexts. This change will support deploying instances from both the `main` branch and `release`, simplifying future implementations and speeding up the process.

Changes:
- Use a single bash script (`gcp-get-cached-disks.sh`) to get cached states names and availability
- Move script logic from `sub-find-cached-disks.yml` to `gcp-get-cached-disks.sh` and adapt `sub-find-cached-disks.yml` to allow to output available disks and disks names.
- Simplify parameters usage in `sub-deploy-integration-tests-gcp.yml` and convert the `Find ${{ inputs.test_id }} cached state disk` step into an independent job, to be able to use the `sub-find-cached-disks.yml` reusable workflow
- Remove repetition in `sub-ci-integration-tests-gcp.yml`
We had a technical debt with some tests using a hardcoded value for the cache directory (`/zebrad-cache`), which generated inconsistency across disks and cached states directories.

Changes:
- Allow sync tests to use the `ZEBRA_CACHED_STATE_DIR` as the cache directory, if specified
- Update the `entrypoint.sh` to reflect this change
- Add the `ZEBRA_CACHED_STATE_DIR` variable to the missing tests in `sub-ci-integration-tests-gcp.yml`, and remove extra parameters to call reusable workflows.
@gustavovalverde gustavovalverde added A-infrastructure Area: Infrastructure changes A-devops Area: Pipelines, CI/CD and Dockerfiles I-usability Zebra is hard to understand or use C-feature Category: New features C-tech-debt Category: Code maintainability issues P-High 🔥 labels Sep 17, 2024
@gustavovalverde gustavovalverde self-assigned this Sep 17, 2024
@github-actions github-actions bot added the C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG label Sep 17, 2024
@gustavovalverde gustavovalverde changed the title feat(cd): deploy instances with cached states feat(cd): deploy instances with attached cached states Sep 17, 2024
@gustavovalverde gustavovalverde marked this pull request as ready for review September 17, 2024 18:34
@gustavovalverde gustavovalverde requested a review from a team as a code owner September 17, 2024 18:34
@gustavovalverde gustavovalverde requested review from oxarbitrage and removed request for a team September 17, 2024 18:34
oxarbitrage
oxarbitrage previously approved these changes Sep 19, 2024
Base automatically changed from ref-cache-state to main September 19, 2024 12:31
@mergify mergify bot dismissed oxarbitrage’s stale review September 19, 2024 12:31

The base branch was changed.

@mergify mergify bot requested a review from a team as a code owner September 19, 2024 12:31
@mergify mergify bot requested review from oxarbitrage and removed request for a team September 19, 2024 12:31
mergify bot added a commit that referenced this pull request Sep 19, 2024
@mergify mergify bot merged commit 8870b2c into main Sep 19, 2024
110 checks passed
@mergify mergify bot deleted the feat-deploy-on-main branch September 19, 2024 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-devops Area: Pipelines, CI/CD and Dockerfiles A-infrastructure Area: Infrastructure changes C-feature Category: New features C-tech-debt Category: Code maintainability issues C-trivial Category: A trivial change that is not worth mentioning in the CHANGELOG I-usability Zebra is hard to understand or use P-High 🔥
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use cached state for continuous deployment instances
2 participants