feat: blob implementation #1342
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vasco-santos
force-pushed
the
feat/blob-implementation
branch
2 times, most recently
from
d3c09d2 to
7660794
Compare
vasco-santos
force-pushed
the
feat/blob-implementation
branch
from
7660794 to
52cf8fe
Compare
vasco-santos
commented
Mar 25, 2024
vasco-santos
force-pushed
the
feat/blob-implementation
branch
from
52cf8fe to
02584d7
Compare
vasco-santos
commented
Mar 25, 2024
vasco-santos
force-pushed
the
feat/blob-implementation
branch
3 times, most recently
from
472a097 to
9624c8a
Compare
10 tasks
vasco-santos
force-pushed
the
feat/blob-implementation
branch
10 times, most recently
from
6178240 to
a78c3ea
Compare
vasco-santos
force-pushed
the
feat/blob-implementation
branch
3 times, most recently
from
a36b64d to
5c6b000
Compare
Gozala
reviewed
Apr 5, 2024
There was a problem hiding this comment.
This is an amazing progress, I'm blown away. I did provide some feedback here and there only thing I'd call out is that we really should capture dependencies across tasks and we'd have to define those in capability schemas also otherwise they will be ignored during encode.
Other than that I was also thinking when writing spec that it would be a good idea to have separate internal invocations when space usage is increased / decreased so that billing could more simply tap into that without having to make guesses if allocate resulted in more space use or not. Relatedly there had been various other cases where we want to do batch operations that billing would have trouble with, however it could be also addressed if we had simple internal ops that increase / decreate usage. This is definitely something we need @alanshaw feedback on
Comment on lines
84
to
99
| if (hasBlob.ok) { | ||
| return { | ||
| ok: { size: blob.size }, | ||
| } | ||
| } | ||
|
|
||
| return { | ||
| ok: { | ||
| size: blob.size, | ||
| address: { | ||
| url: createUploadUrl.ok.url.toString(), | ||
| headers: createUploadUrl.ok.headers, | ||
| }, | ||
| }, | ||
| } | ||
| }) |
There was a problem hiding this comment.
I would expect allocation to produce same result regardless of whether we have a blob or not, it's just if we do have a blob already we would succeed put without awaiting on user to perform upload and consequently succeed blob/add as well.
There was a problem hiding this comment.
I am following the spec, which makes address optional for this reason? https://github.com/web3-storage/specs/blob/feat/http-put/w3-blob.md#allocate-blob-receipt-schema
In the meantime, latest implementation of blob dd should actually do what you say
vasco-santos
force-pushed
the
feat/blob-implementation
branch
3 times, most recently
from
6f1b921 to
0668957
Compare
vasco-santos
force-pushed
the
feat/blob-implementation
branch
from
adfb56f to
15d732c
Compare
This was referenced Apr 12, 2024
vasco-santos
pushed a commit
that referenced
this pull request
Apr 12, 2024
🤖 I have created a release *beep* *boop* --- ## [13.3.0](capabilities-v13.2.1...capabilities-v13.3.0) (2024-04-12) ### Features * blob, web3.storage and ucan conclude capabilities together with api handlers ([#1342](#1342)) ([00735a8](00735a8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
vasco-santos
added a commit
that referenced
this pull request
Apr 12, 2024
🤖 I have created a release *beep* *boop* --- ## [9.1.0](upload-api-v9.0.1...upload-api-v9.1.0) (2024-04-12) ### Features * blob, web3.storage and ucan conclude capabilities together with api handlers ([#1342](#1342)) ([00735a8](00735a8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: Vasco Santos <[email protected]>
alanshaw
pushed a commit
to storacha/upload-service
that referenced
this pull request
Nov 5, 2024
🤖 I have created a release *beep* *boop* --- ## 1.0.0 (2024-11-05) ### ⚠ BREAKING CHANGES * add `index/add` handler ([storacha#1421](https://github.com/storacha/upload-service/issues/1421)) * restrict store API to CARs ([storacha#1415](https://github.com/storacha/upload-service/issues/1415)) * **capabilities:** `BlobMultihash` type in `@web3-storage/capabilities` renamed to `Multihash`. * allocations storage interface now requires remove to be implemented * return allocated bytes in `store/add` receipt ([storacha#1213](https://github.com/storacha/upload-service/issues/1213)) * coupon ([storacha#1136](https://github.com/storacha/upload-service/issues/1136)) * see latest specs https://github.com/web3-storage/specs/blob/cbdb706f18567900c5c24d7fb16ccbaf93d0d023/w3-filecoin.md * filecoin client to use new capabilities * filecoin capabilities ### refactor * filecoin api services events and tests ([storacha#974](https://github.com/storacha/upload-service/issues/974)) ([953537b](953537b)) * filecoin capabilities ([c0b97bf](c0b97bf)) * filecoin client to use new capabilities ([b0d9c3f](b0d9c3f)) ### Features * add "plan/create-admin-session" capability ([storacha#1411](https://github.com/storacha/upload-service/issues/1411)) ([50eeeb5](50eeeb5)) * add `index/add` handler ([storacha#1421](https://github.com/storacha/upload-service/issues/1421)) ([cbe9524](cbe9524)) * add `initialize` method to `PlansStorage` ([storacha#1278](https://github.com/storacha/upload-service/issues/1278)) ([6792126](6792126)) * add `store/get` and `upload/get` capabilities ([storacha#942](https://github.com/storacha/upload-service/issues/942)) ([40c79eb](40c79eb)) * add `subscription/list` capability ([storacha#1088](https://github.com/storacha/upload-service/issues/1088)) ([471d7e5](471d7e5)) * add a function to verify and return Abilities. ([storacha#1252](https://github.com/storacha/upload-service/issues/1252)) ([2f026a2](2f026a2)) * add blob list and remove ([storacha#1385](https://github.com/storacha/upload-service/issues/1385)) ([2f69946](2f69946)) * add blob protocol to upload-client ([storacha#1425](https://github.com/storacha/upload-service/issues/1425)) ([49aef56](49aef56)) * add blob/get ([storacha#1484](https://github.com/storacha/upload-service/issues/1484)) ([328039d](328039d)) * add revocation to access-client and w3up-client ([storacha#975](https://github.com/storacha/upload-service/issues/975)) ([6c877aa](6c877aa)) * add usage/report capability ([storacha#1079](https://github.com/storacha/upload-service/issues/1079)) ([6418b4b](6418b4b)) * blob, web3.storage and ucan conclude capabilities together with api handlers ([storacha#1342](https://github.com/storacha/upload-service/issues/1342)) ([00735a8](00735a8)) * **capabilities:** add `index/add` capability ([storacha#1410](https://github.com/storacha/upload-service/issues/1410)) ([1b71b89](1b71b89)) * change `plan/update` to `plan/set` and use existing `PlansStorage#set` to implement an invocation handler ([storacha#1258](https://github.com/storacha/upload-service/issues/1258)) ([1ccbfe9](1ccbfe9)) * coupon ([storacha#1136](https://github.com/storacha/upload-service/issues/1136)) ([1b94f2d](1b94f2d)) * filecoin info ([storacha#1091](https://github.com/storacha/upload-service/issues/1091)) ([adb2442](adb2442)) * Generate Space proofs on the fly, on `access/claim` ([storacha#1555](https://github.com/storacha/upload-service/issues/1555)) ([9e2b1d4](9e2b1d4)) * implement `plan/get` capability ([storacha#1005](https://github.com/storacha/upload-service/issues/1005)) ([f0456d2](f0456d2)) * introduce capability for changing billing plan ([storacha#1253](https://github.com/storacha/upload-service/issues/1253)) ([d33b3a9](d33b3a9)) * move aggregate information out of deals in filecoin/info ([storacha#1192](https://github.com/storacha/upload-service/issues/1192)) ([18dc590](18dc590)) * publish index claim ([storacha#1487](https://github.com/storacha/upload-service/issues/1487)) ([237b0c6](237b0c6)) * restrict store API to CARs ([storacha#1415](https://github.com/storacha/upload-service/issues/1415)) ([e53aa87](e53aa87)) * return allocated bytes in `store/add` receipt ([storacha#1213](https://github.com/storacha/upload-service/issues/1213)) ([5d52e44](5d52e44)) * router ([#11](#11)) ([c810735](c810735)) * upgrade ucanto/transport to 9.1.0 in all packages to get more verbose errors from HTTP transport on non-ok response ([storacha#1312](https://github.com/storacha/upload-service/issues/1312)) ([d6978d7](d6978d7)) * usage/record capability definition ([storacha#1562](https://github.com/storacha/upload-service/issues/1562)) ([98c8a87](98c8a87)) * wip router ([ffcd9c7](ffcd9c7)) ### Fixes * add missing ContentNotFound definition for filecoin offer failure ([c0b97bf](c0b97bf)) * add missing filecoin submit success and failure types ([c0b97bf](c0b97bf)) * capabilities should export blob caps ([storacha#1376](https://github.com/storacha/upload-service/issues/1376)) ([460729e](460729e)) * client tests ([b0d9c3f](b0d9c3f)) * fix arethetypesworking errors in all packages ([storacha#1004](https://github.com/storacha/upload-service/issues/1004)) ([2e2936a](2e2936a)) * issue where typedoc docs would only show full docs for w3up-client ([storacha#1141](https://github.com/storacha/upload-service/issues/1141)) ([0b8d3f3](0b8d3f3)) * migrate repo ([storacha#1389](https://github.com/storacha/upload-service/issues/1389)) ([475a287](475a287)) * one more tweak to the `PlanStorage` interface ([storacha#1280](https://github.com/storacha/upload-service/issues/1280)) ([5a44565](5a44565)) * package metadata ([storacha#1161](https://github.com/storacha/upload-service/issues/1161)) ([b8a1cc2](b8a1cc2)) * put access.session back ([storacha#1100](https://github.com/storacha/upload-service/issues/1100)) ([10a1a4b](10a1a4b)) * rename blob and index client capabilities ([storacha#1478](https://github.com/storacha/upload-service/issues/1478)) ([17e3a31](17e3a31)) * repo URLs ([storacha#1550](https://github.com/storacha/upload-service/issues/1550)) ([e02ddf3](e02ddf3)) * tests ([b179910](b179910)) * trigger capabilities release ([storacha#1399](https://github.com/storacha/upload-service/issues/1399)) ([7d9ab35](7d9ab35)) * type errors ([c0b97bf](c0b97bf)) * update data-segment dep ([228ff79](228ff79)) * upgrade @ucanto/validator with bugfix ([storacha#1151](https://github.com/storacha/upload-service/issues/1151)) ([d4e961b](d4e961b)) * upgrade ucanto core ([storacha#1127](https://github.com/storacha/upload-service/issues/1127)) ([5ce4d22](5ce4d22)) * upgrade ucanto in filecoin api ([c95fb54](c95fb54)) * upgrade ucanto libs and format filecoin api ([storacha#1359](https://github.com/storacha/upload-service/issues/1359)) ([87ca098](87ca098)) * upload API test fixes ([6b0d72d](6b0d72d)) ### Other Changes * Add `pnpm dev` to watch-build all packages ([storacha#1533](https://github.com/storacha/upload-service/issues/1533)) ([07970ef](07970ef)) * **main:** release capabilities 10.1.0 ([storacha#979](https://github.com/storacha/upload-service/issues/979)) ([bdd3970](bdd3970)) * **main:** release capabilities 10.2.0 ([storacha#983](https://github.com/storacha/upload-service/issues/983)) ([a906488](a906488)) * **main:** release capabilities 11.0.0 ([storacha#994](https://github.com/storacha/upload-service/issues/994)) ([76b0489](76b0489)) * **main:** release capabilities 11.0.1 ([storacha#1008](https://github.com/storacha/upload-service/issues/1008)) ([37cdc5a](37cdc5a)) * **main:** release capabilities 11.1.0 ([storacha#1026](https://github.com/storacha/upload-service/issues/1026)) ([7fdb541](7fdb541)) * **main:** release capabilities 11.2.0 ([storacha#1084](https://github.com/storacha/upload-service/issues/1084)) ([0e7b6dc](0e7b6dc)) * **main:** release capabilities 11.3.0 ([storacha#1098](https://github.com/storacha/upload-service/issues/1098)) ([7d671bd](7d671bd)) * **main:** release capabilities 11.3.1 ([storacha#1101](https://github.com/storacha/upload-service/issues/1101)) ([20b5b35](20b5b35)) * **main:** release capabilities 11.4.0 ([storacha#1105](https://github.com/storacha/upload-service/issues/1105)) ([1b6210f](1b6210f)) * **main:** release capabilities 11.4.1 ([storacha#1131](https://github.com/storacha/upload-service/issues/1131)) ([a5b7154](a5b7154)) * **main:** release capabilities 12.0.0 ([storacha#1137](https://github.com/storacha/upload-service/issues/1137)) ([bb23e9f](bb23e9f)) * **main:** release capabilities 12.0.1 ([storacha#1147](https://github.com/storacha/upload-service/issues/1147)) ([d1a9c78](d1a9c78)) * **main:** release capabilities 12.0.2 ([storacha#1152](https://github.com/storacha/upload-service/issues/1152)) ([b9d7ff5](b9d7ff5)) * **main:** release capabilities 12.0.3 ([storacha#1163](https://github.com/storacha/upload-service/issues/1163)) ([ec5c385](ec5c385)) * **main:** release capabilities 12.1.0 ([storacha#1195](https://github.com/storacha/upload-service/issues/1195)) ([a21c1a5](a21c1a5)) * **main:** release capabilities 13.0.0 ([storacha#1230](https://github.com/storacha/upload-service/issues/1230)) ([3d5b3ef](3d5b3ef)) * **main:** release capabilities 13.1.0 ([storacha#1257](https://github.com/storacha/upload-service/issues/1257)) ([85adc9a](85adc9a)) * **main:** release capabilities 13.1.1 ([storacha#1283](https://github.com/storacha/upload-service/issues/1283)) ([31c38e9](31c38e9)) * **main:** release capabilities 13.2.0 ([storacha#1315](https://github.com/storacha/upload-service/issues/1315)) ([0505458](0505458)) * **main:** release capabilities 13.2.1 ([storacha#1362](https://github.com/storacha/upload-service/issues/1362)) ([26b5751](26b5751)) * **main:** release capabilities 13.3.0 ([storacha#1366](https://github.com/storacha/upload-service/issues/1366)) ([d6fbc4a](d6fbc4a)) * **main:** release capabilities 13.3.1 ([storacha#1377](https://github.com/storacha/upload-service/issues/1377)) ([149f592](149f592)) * **main:** release capabilities 14.0.0 ([storacha#1386](https://github.com/storacha/upload-service/issues/1386)) ([69bfc08](69bfc08)) * **main:** release capabilities 14.0.1 ([storacha#1395](https://github.com/storacha/upload-service/issues/1395)) ([a76c970](a76c970)) * **main:** release capabilities 14.0.2 ([storacha#1400](https://github.com/storacha/upload-service/issues/1400)) ([7b46852](7b46852)) * **main:** release capabilities 15.0.0 ([storacha#1412](https://github.com/storacha/upload-service/issues/1412)) ([ec90b81](ec90b81)) * **main:** release capabilities 16.0.0 ([storacha#1419](https://github.com/storacha/upload-service/issues/1419)) ([50e3934](50e3934)) * **main:** release capabilities 17.0.0 ([storacha#1428](https://github.com/storacha/upload-service/issues/1428)) ([6ee21fd](6ee21fd)) * **main:** release capabilities 17.1.0 ([storacha#1447](https://github.com/storacha/upload-service/issues/1447)) ([8ee1fd9](8ee1fd9)) * **main:** release capabilities 17.1.1 ([storacha#1483](https://github.com/storacha/upload-service/issues/1483)) ([5394ed5](5394ed5)) * **main:** release capabilities 17.2.0 ([storacha#1494](https://github.com/storacha/upload-service/issues/1494)) ([99876a5](99876a5)) * **main:** release capabilities 17.3.0 ([storacha#1503](https://github.com/storacha/upload-service/issues/1503)) ([891c2a5](891c2a5)) * **main:** release capabilities 17.4.0 ([storacha#1559](https://github.com/storacha/upload-service/issues/1559)) ([feaea7a](feaea7a)) * no longer depends on hd-scripts, packages use/configure eslint directly, fixes warnings from npm lint script ([storacha#1058](https://github.com/storacha/upload-service/issues/1058)) ([ebdb99b](ebdb99b)) * package renames ([0f797ed](0f797ed)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds implementation of
blob/*,web3.storage/*anducan/concludehandlers and capabilities.Also upgrades to latest ucanto all over the place