[parent] write to destinations that don't support bucket events

[alanshaw]

What

This ticket describes work to that removes dependence on S3 bucket events in our pipeline, and ultimately switches to having our uploads go directly to R2. As a side effect of this work, we'll be able to support uploads to any storage device that can support a rough object storage interface. For our users, this increases our platforms resiliency to outages because we're no longer locked into a specific platform.

• [parent] blob protocol #17
• [parent] filecoin/offer flow #18
• [parent] ipni/offer protocol #54
• Identify migration plan to insure smooth upgrade path for clients. #25
• Manual testing and deploy of write to anywhere #83
• move capabilities to namespace that's already delegated #84
• Propogate new w3up to cli #93
• Propogate new w3up to console #94
• Check to make sure no store/* references in docs and update if neccesary. #95

Next:

• FIlecoin follow up filecoin-api storefront reads Blob bytes to derive Piece CID from anywhere w3up#1349

[heyjay44]

From old ticket:

• IPNI provider claims. Status: in progress (was Oli)
• roundabout PR in review. Status: in review (was Vasco)
• hoverboard PR. Status: in progress (was Bengo)

[vasco-santos]

Context ticket https://hackmd.io/n7tKbl0GSPKrEnWDv1Sa0Q?view

[hannahhoward]

@heyjay44 I do not believe any of the above tickets are ultimately part of the above are ultimately part of this endeavor, but rather are related to work to remove carpark

[reidlw]

@vasco-santos - could you please provide an updated status on this overall workstream her in the ticket? Thanks!

[vasco-santos]

Last week the highlights were:

1. [parent] filecoin/offer flow #18 - has 2 PRs + Rollout plan
   1.1. a PR for filecoin-api that makes the submit flow validate the Piece CID offered by client + issue equivalency claim of CarCID to Piece CID
   1.2. an issue with rollout plan for these changes and how to support (until a given date) old clients not invoking filecoin/offer
   1.3. (blocked on PR above) a PR for w3infra integrating above changes in the infra + changing bucket event current behaviour of computing piece and moving it along the w3filecoin pipeline to actually simulate client filecoin/offer in store/add bucket flow
   1.4. client invokes filecoin/offer by default PR created/merged and upload-client + w3up-client were released with this
   1.5. Note that when above PRs are merged, we are close to get the filecoin thing in place. But there is some small follow up work as we need to integrate location claims in the API (which is pending things to get done on the points below, and likely I will create a small ticket for it when above is merged)
2. Datawherehouse - RFC got merged, and based on discussions there + slack thread and sync time we opted to move forward with blob protocol (wrapping datawherehouse within it)
3. [parent] blob protocol #17 we aimed at first to hook datawherehouse+store/publish within store/* current flow. But that would make it difficult to do a rollout where old clients would still perform store/add but not compute and publish everything we aim client to start to do. At the same time, @Gozala at previously proposed that we should have a blob/* protocol to replace store/* so that we would not be tied to CAR files like store/* assumes. We ended up expanding the scope of this project to also include blob/* given it is not a huge lift and we can already hook up new clients to blob/* invocations, while keeping support for current behaviour with store/* for older clients for some more time
   3.1. @Gozala created a blob Protocol spec that we iterated together and got merged last week feat: blob protocol draft specs#115 - there is still some work to do there, but we are happy with initial version
   3.2. Started implementing blob protocol capabilities and server handlers that will continue throughout next week
   3.3. We are aligning on some impl details on A stop gap solution for blob implementation until invocation spec w3up#1339

[vasco-santos]

We currently have a blocker on decision for the implementation of blob/* protocol.

📚 I wrote an issue about an important rollout decision that needs to happen for landing blob/* protocol. More specifically, how we integrate with old store/* protocol at the state/data level: storacha-network/w3up#1343

• for some additional context important for decision, on the past
  @dchoi27
  was discussing with PL whether historical buckets would be on PL bill on nucleation (and I think from what he one time wrote in zoom chat, PL agreed this?). This was one of the main triggers to get the write to anywhere thing, as we wanted to be able to write to a new bucket on nucleation

@hannahhoward @reidlw @alanshaw need your input and help to get to a decision in the issue storacha-network/w3up#1343

[vasco-santos]

Small update from last 2 days:

1. looks like there is alignment on What integration to have between Store and Blob protocol implementations w3up#1343 to not worry about deduping for now
2. filecoin-api was released with server support for non bucket event, needs now PR with infra updates to be reviewed/released. See [parent] filecoin/offer flow #18 (comment)
3. blob implementation is in progress feat: blob implementation w3up#1342
4. @Gozala started the convo on what we need for IPNI on Support IPNI+Carpark indexing from the client #24

[vasco-santos]

Status update for the end of week:

1. @alanshaw @Gozala and I met and decided to move away from double blob/add to report thing is done to something that is more aligned to "user sends a signed receipt that thing is done" by introducing a http/put capability for the client to create a receipt, as well as a ucan/receipt capability that service provides to receive receipts.

• @Gozala will create PR to update the spec

2. Blob implementation is on the way feat: blob implementation w3up#1342

• What is done?
  • capabilities are defined
  • handlers for blob/add and web3.storage/blob/allocate are implemented and tested
  • handler for web3.storage/blob/accept needs to be finished and tested
• What is next?
  • @vasco-santos to implement decisions from point 1, and with that finalize accept part
  • @vasco-santos to implement and spec list + remove (maybe get out of scope for first iteration removing given claim revocation, or do not do claims for now)

3. we are getting alignment on Support IPNI+Carpark indexing from the client #24
4. @Gozala will review feat: upgrade storefront api with filecoin offer from bucket event instead of directly compute w3infra#343