Indexing design for the future W3S Network

[hannahhoward]

What
A comprehensive design for how we index and find content in the future W3S network.

Why
Having a clear design and implementation for indexing is a blocker for even having a version of web3.storage that is a network of storage nodes.

We've realized that the W3 IPNI Indexing RFC is unlikely to a complete solution to indexing our content.

In particular, we are concerned about:

1. Relying on public IPNI that is maintained outside the team completely, such that the whole system would go down if IPNI failed.
2. Performance on reads for our current IPNI design
3. Most critically, satisfying a gaurantee of read on write (i.e. when a user finishes an upload of any kind, they can be assurred that their content is immediately readable).

We've discussed various solutions including:

1. Running a private instance of IPNI, possibly with modifications
2. Designing a new index for our own content that provides transcational gaurantees and fewer round trips for read performance

Ultimately, the most key distinction is public vs private indexing

• We want to publish our data to public indexers like IPNI, and may even rely on those indexes for read protocols we're not heavily optimizing or investing in -- like Bitswap
• We still need a high availability, very fast way to publish and quickly find content in our network.

Cost
The design for this endeavor should be completable in a sprint, assuming a collaboration between a few key team members (@gammazero, @Gozala for example)

[Gozala]

Sketch of the design https://www.tldraw.com/s/v2_c_n4nrJnxVuCLFdmJEo6H36?v=275%2C287%2C2344%2C1755&p=page

[hannahhoward]

storacha/RFC#29 - needs review
storacha/RFC#22 - needs review