feat(ipld): pass bindnode opts #31
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adds the ability to pass opts to schema structs so we can avoid the intermediate map step in many cases
hannahhoward
force-pushed
the
feat/pass-bindnode-opts
branch
from
8c5af08 to
d4cff4a
Compare
This was referenced Nov 1, 2024
Merged
alanshaw
approved these changes
Nov 4, 2024
hannahhoward
force-pushed
the
feat/pass-bindnode-opts
branch
from
b79cc7b to
aa8566a
Compare
hannahhoward
added a commit
to storacha/go-capabilities
that referenced
this pull request
Nov 11, 2024
# Goals Utilize storacha/go-ucanto#31 to make this whole repo a whole lot simpler # Implementation All of the verbose ToIPLD and schema.Mapped functions can basically be cut, along with all the model objects, by simply utilizing schema options (aka bindnode options)
hannahhoward
added a commit
to storacha/storage
that referenced
this pull request
Nov 11, 2024
# Goals Implement PDP through curio # Implementation The basic strategy here is as follows: 1. When PDP is present, it replaces a number of parts of the blob service, as they now happen directly through curio. (I actually toyed with mirroring the Access and Presigner APIs but it was a little awkward 1. The big chunk of new logic is the aggregator, which is kind of like a mini-w3filecoin pipeline but a lot simpler. I've built this with the goal of making it runnable on SQS queues just like the w3filecoin pipeline, or runnable locally with the job queue. The aggregator can be viewed in terms of its constituent packages: 1. `pkg/pdp/aggregator/aggregate` handles calculating aggregate roots and assembling piece proofs (it's like a mini go-data-segment 1. `pkg/pdp/aggregator/fns` contains the core logic of each aggregation step, minutes side effects/state 1. `pkg/pdp/aggregator/steps.go` contains the actual steps in aggregation with side effects 1. `pkg/pdp/aggregator/local.go` contains the implementation of aggregation that can run on a native piece of hardware. AWS version to come later 1. PieceAdder is another function that handles essentially saying to Curio "I have a new piece to upload" and Curio responds with an upload URL -- it's relatively analogous to our blob/allocate actually (except we do it in this node) 1. PieceFinder gets the piece cid from the blob hash and size. Importantly, this is probably the ugliest bit of logic for now cause Curio isn't currently setup for reliable read on write, so there is a change we will get blob/accept BEFORE we Curio is ready to give us the piece cid -- that's why there's the retry loop in there for now 1. `pkg/pdp/curio` is just the curio client ot make http calls to curio 1. I also implemented the ReceiptStore cause I needed it 1. I also added a utility for storing ipld types in a data store -- see `pkg/internal/ipldstore` -- I think this will be useful in future 1. There are essentially one new invocation available on the UCAN endpoint - `pdp/info` which you can use to get the state of your aggregation and the inclusion proof. This may still need some changes on auth though? Not sure 1. `pdp/accept` is a self issued invocation used by the server to track when PDP is done and save results. The PR depends on the following: storacha/go-ucanto#31 storacha/go-capabilities#2 storacha/go-capabilities#3 as well as two new repos: https://github.com/storacha/go-piece https://github.com/storacha/go-jobqueue (extracted from indexing-service) --------- Co-authored-by: ash <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Goals
Improve ergonomics of working with schemas and structs with bindnode options
Implementation
So here's the thing: I hade the Model extra copies being everywhere. It's fine when it's a complex class wrapping a model like in receipt, but a LOT of boilerplate when we're dealing with value objects -- namely capabilities and ok / err types on results. The thing is, most of these value objects can be used directly instead of having second version if they had just a little more logic around conversion from schemas. Fortunately, IPLD's bindnode actually has that through bindnode options, which while they're marked experimental, have been around for years and @rvagg just accepted my PR to make them work on named schema types, which means you can do a LOT with them.
So, this PR basically just passes around bindnode opts to everything so you can use them (they're all var args). And, it adds some helper versions of bindnode opts that work with generics that are more ergonomic to use.