Publish indexed transactions with BLAKE2b hashes to IPFS DHT #10468

dmitry-markin · 2025-11-28T16:17:29Z

Add --ipfs-bootnodes flag for specifying IPFS bootnodes. If passed along with --ipfs-server, the node will register as a content provider in IPFS DHT of indexed transactions with BLAKE2b hashes of the last two weeks (or pruning depth if smaller).

dmitry-markin · 2025-11-28T16:49:30Z

/cmd prdoc --audience node_dev --bump major

…e_dev --bump major'

dmitry-markin · 2025-11-28T16:58:00Z

substrate/client/network/src/ipfs_block_provider.rs

This file is taken from https://github.com/zdave-parity/substrate/blob/polkadot-bulletin-chain/client/network/src/ipfs/block_provider.rs The code is checked for correctness, but still should be reviewed.

Big thanks goes to @zdave-parity for solid implementation!

lexnv · 2025-12-03T15:36:17Z

cumulus/client/pov-recovery/src/tests.rs

 	}

-	fn indexed_transaction(&self, _: Block::Hash) -> sp_blockchain::Result<Option<Vec<u8>>> {
+	fn indexed_transaction(&self, _: H256) -> sp_blockchain::Result<Option<Vec<u8>>> {


dq: Does the Block::Hash resolve to something different here? Or we should update the other methods of the trait?

Block::Hash can be any hash internally, including hashes of different size. H256 is a fixed-length 32-byte hash, in this case BLAKE2b-256.

I am glad that you touched on these topics. Please check my thoughts from the previous issue, “IPFS/Bitswap CID filtering” #10538 — one of them is exactly about this hash/block_hash/content_hash question.

lexnv · 2025-12-03T15:38:37Z

substrate/client/api/src/client.rs

+	///
+	/// Note that this will only fetch transactions that are indexed by the runtime with
+	/// `storage_index_transaction`.
+	fn block_indexed_hashes(&self, hash: Block::Hash) -> sp_blockchain::Result<Option<Vec<H256>>>;


dq: Should we keep generic over the returned values (ie replace H256 with BlockT::Hash)? (I do like the h256 better, asking for consistency)

This is also a BLAKE2b-256 and not a block hash.

lexnv · 2025-12-03T15:43:56Z

substrate/client/db/src/lib.rs

+		&self,
+		hash: Block::Hash,
+	) -> ClientResult<Option<impl Iterator<Item = DbHash>>> {
+		let body = match read_db(


nit: let Some(body) = read_db else { return Ok(None)};

lexnv · 2025-12-03T15:45:02Z

substrate/client/db/src/lib.rs

+							"Missing indexed transaction {hash:?}",
+						))),
+					})
+					.collect::<Result<_, _>>()?,


nit: The Result type is needed by the compiler because of adding the sp_blockchain::Error?

Result is needed to resolve the first Err yielded by the iterator as an Err of entire sequence.

lexnv · 2025-12-03T15:45:58Z

substrate/client/network/src/litep2p/bitswap.rs


 /// Logging target for the file.
-const LOG_TARGET: &str = "sub-libp2p::bitswap";
+const LOG_TARGET: &str = "sub-libp2p::ipfs::bitswap";


nit: I would keep sub-libp2p::bitswap since grafana does support only 1 subtarget

lexnv · 2025-12-03T15:49:55Z

substrate/client/network/src/litep2p/ipfs_dht.rs

+
+		loop {
+			tokio::select! {
+				change = changes.next(), if inflight_queries.len() < MAX_INFLIGHT_QUERIES => {


nit: We could add tiny handler methods for these code blocks. Currently code under tokio::select! is not formatted:

change = changes.next() => { self.handle_change(change: Change) } fn handle_change(self, change: Change) { if inflight_queries.len() >= MAX { return; } }

This is not the same, because , if ... in tokio::select! pauses polling the stream and not just discards the Change. But yeah, I will look into formatting the select! in a nicer way.

lexnv · 2025-12-03T15:50:47Z

substrate/client/network/src/litep2p/ipfs_dht.rs

+					match change {
+						None => {
+							debug!(target: LOG_TARGET, "BlockProvider terminated, terminating IpfsDht");
+							return


nit: Should we exit sooner here regardless of if inflight_queries.len() < MAX_INFLIGHT_QUERIES?

It will delay the exit (requiring the query slot to empty), but in the end this is abnormal termination and won't make a big difference of when we exit.

lexnv · 2025-12-03T15:51:44Z

substrate/client/network/src/litep2p/ipfs_dht.rs

+						continue;
+					}
+
+					random_walk_query = Some(self.kademlia_handle.find_node(PeerId::random()).await);


Is 1 query at a time sufficient to discover the network? I believe this is how libp2p works, but IRRC litep2p does quite a few more

In practice it works even without random walks :)

lexnv · 2025-12-03T15:54:51Z

substrate/client/network/src/litep2p/ipfs_dht.rs

+							if Some(query_id) == random_walk_query {
+								trace!(target: LOG_TARGET, "DHT random walk failed");
+
+								random_walk_query = None;


For the following example, we'll wait quite a bit without issuing a new query:

T0: query submited

T1 (10seconds later): query fails

T2: (9 minutes 50 seconds later): we resubmit the query

Could we initiate the random walk on this failure instead? This way we always have an inflight query until we can find peers

I can think only about one situation when query fails — no network connectivity. In other cases it is enough for one peer to respond to FIND_NODE for query to yield some peers. In case of no network connectivity initiating the query instantly on failure will lead to busy looping. And, in the end, it is not that important that we "miss" the random walk, as it is not super important (we can do these random walks once per hour with likely the same outcome).

lexnv · 2025-12-03T15:55:57Z

substrate/client/network/src/config.rs

+	/// Enable publishing of indexed transactions to IPFS.
 	pub ipfs_server: bool,

+	/// List of IPFS bootstrap nodes.


nit: We could mention that only direct requests will be handled here if no bootnods are provided

lexnv · 2025-12-03T16:05:33Z

substrate/client/network/src/ipfs_block_provider.rs

+			// nothing to do.
+			debug_assert!(this.fetched_to <= this.finalized_to);
+			if this.fetched_to == this.finalized_to {
+				return Poll::Pending


Since we are not saving the waker, we rely entirely on this.finality_notifications.poll_next_unpin(cx) to wake us up, is that intended?

Yes, this is fine — we come here only if finality_notifications yielded Poll::Pending, so it will wake us up.

lexnv

LGTM! 🙏

dmitry-markin added 9 commits November 19, 2025 18:06

Add zdave-parity's BlockProvider with zero other changes

f12fc2c

Use correct type for indexed transaction hashes

b2411ad

Add BlockBackend::block_indexed_hashes

d6c06f6

Bump litep2p 0.12.0 -> 0.12.1

9a963c7

Prepare IPFS config

f2af1ea

minor: move BlockProvider to sc-network root

f2386bd

Implement IPFS DHT publisher

294ea60

Increase Kademlia memory store size

2b999d1

Add DHT random walks

3861b6d

dmitry-markin requested review from bkontur and lexnv November 28, 2025 16:17

dmitry-markin self-assigned this Nov 28, 2025

dmitry-markin added the T0-node This PR/Issue is related to the topic “node”. label Nov 28, 2025

dmitry-markin added this to Networking Nov 28, 2025

Update from github-actions[bot] running command 'prdoc --audience nod…

5d08673

…e_dev --bump major'

dmitry-markin commented Nov 28, 2025

View reviewed changes

dmitry-markin added 2 commits November 28, 2025 19:38

Update prdoc

c0daab0

minor: remove useless async block

e27395d

dmitry-markin mentioned this pull request Nov 28, 2025

Add IPFS Amino DHT support paritytech/polkadot-bulletin-chain#33

Open

7 tasks

dmitry-markin added 4 commits November 28, 2025 21:14

Update prdoc

4294ffb

Limit in-flight Kademlia queries

deeeb18

minor: typo

392f5be

Merge remote-tracking branch 'origin/master' into dm-ipfs-dht

626760b

lexnv reviewed Dec 3, 2025

View reviewed changes

lexnv approved these changes Dec 3, 2025

View reviewed changes

bkontur mentioned this pull request Dec 4, 2025

[IPFS/Bitswap] Improvement Suggestions #10538

Open

Publish indexed transactions with BLAKE2b hashes to IPFS DHT #10468

Are you sure you want to change the base?

Publish indexed transactions with BLAKE2b hashes to IPFS DHT #10468

Uh oh!

Conversation

dmitry-markin commented Nov 28, 2025

Uh oh!

dmitry-markin commented Nov 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lexnv Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmitry-markin Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmitry-markin Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lexnv left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lexnv Dec 3, 2025 •

edited

Loading

dmitry-markin Dec 3, 2025 •

edited

Loading

dmitry-markin Dec 3, 2025 •

edited

Loading