You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What
A comprehensive design for how we index and find content in the future W3S network.
Why
Having a clear design and implementation for indexing is a blocker for even having a version of web3.storage that is a network of storage nodes.
We've realized that the W3 IPNI Indexing RFC is unlikely to a complete solution to indexing our content.
In particular, we are concerned about:
Relying on public IPNI that is maintained outside the team completely, such that the whole system would go down if IPNI failed.
Performance on reads for our current IPNI design
Most critically, satisfying a gaurantee of read on write (i.e. when a user finishes an upload of any kind, they can be assurred that their content is immediately readable).
We've discussed various solutions including:
Running a private instance of IPNI, possibly with modifications
Designing a new index for our own content that provides transcational gaurantees and fewer round trips for read performance
Ultimately, the most key distinction is public vs private indexing
We want to publish our data to public indexers like IPNI, and may even rely on those indexes for read protocols we're not heavily optimizing or investing in -- like Bitswap
We still need a high availability, very fast way to publish and quickly find content in our network.
Cost
The design for this endeavor should be completable in a sprint, assuming a collaboration between a few key team members (@gammazero, @Gozala for example)
The text was updated successfully, but these errors were encountered:
What
A comprehensive design for how we index and find content in the future W3S network.
Why
Having a clear design and implementation for indexing is a blocker for even having a version of web3.storage that is a network of storage nodes.
We've realized that the W3 IPNI Indexing RFC is unlikely to a complete solution to indexing our content.
In particular, we are concerned about:
We've discussed various solutions including:
Ultimately, the most key distinction is public vs private indexing
Cost
The design for this endeavor should be completable in a sprint, assuming a collaboration between a few key team members (@gammazero, @Gozala for example)
The text was updated successfully, but these errors were encountered: