feat(pdp): add PDP capabilties #3
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Merged
alanshaw
approved these changes
Nov 4, 2024
pkg/pdp/pdp.go
Outdated
| pdpTS = ts | ||
| } | ||
|
|
||
| func PDPAcceptCaveatsType() ipldschema.Type { |
There was a problem hiding this comment.
Can we not prefix all types with PDP 😅 🙏
hannahhoward
added a commit
to storacha/storage
that referenced
this pull request
Nov 11, 2024
# Goals Implement PDP through curio # Implementation The basic strategy here is as follows: 1. When PDP is present, it replaces a number of parts of the blob service, as they now happen directly through curio. (I actually toyed with mirroring the Access and Presigner APIs but it was a little awkward 1. The big chunk of new logic is the aggregator, which is kind of like a mini-w3filecoin pipeline but a lot simpler. I've built this with the goal of making it runnable on SQS queues just like the w3filecoin pipeline, or runnable locally with the job queue. The aggregator can be viewed in terms of its constituent packages: 1. `pkg/pdp/aggregator/aggregate` handles calculating aggregate roots and assembling piece proofs (it's like a mini go-data-segment 1. `pkg/pdp/aggregator/fns` contains the core logic of each aggregation step, minutes side effects/state 1. `pkg/pdp/aggregator/steps.go` contains the actual steps in aggregation with side effects 1. `pkg/pdp/aggregator/local.go` contains the implementation of aggregation that can run on a native piece of hardware. AWS version to come later 1. PieceAdder is another function that handles essentially saying to Curio "I have a new piece to upload" and Curio responds with an upload URL -- it's relatively analogous to our blob/allocate actually (except we do it in this node) 1. PieceFinder gets the piece cid from the blob hash and size. Importantly, this is probably the ugliest bit of logic for now cause Curio isn't currently setup for reliable read on write, so there is a change we will get blob/accept BEFORE we Curio is ready to give us the piece cid -- that's why there's the retry loop in there for now 1. `pkg/pdp/curio` is just the curio client ot make http calls to curio 1. I also implemented the ReceiptStore cause I needed it 1. I also added a utility for storing ipld types in a data store -- see `pkg/internal/ipldstore` -- I think this will be useful in future 1. There are essentially one new invocation available on the UCAN endpoint - `pdp/info` which you can use to get the state of your aggregation and the inclusion proof. This may still need some changes on auth though? Not sure 1. `pdp/accept` is a self issued invocation used by the server to track when PDP is done and save results. The PR depends on the following: storacha/go-ucanto#31 storacha/go-capabilities#2 storacha/go-capabilities#3 as well as two new repos: https://github.com/storacha/go-piece https://github.com/storacha/go-jobqueue (extracted from indexing-service) --------- Co-authored-by: ash <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Goals
Define simple invocations for PDP to enable tracking of what happens on a client
Implement
Defines a PDP/accept and PDP/info capability
For Discussion
This refactor currently has a panic in it, and needs modification