feat(upload-client): incremental upload API #727
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…web3-storage/capabilities (#218) Motivation: * #80 To do: * [x] initial package split * [x] remove capabilities dir from packages/access-client and fix imports in access-client * [x] fix imports from other packages that were importing from access-client/capabilities * [x] consider splitting to `capabilities/{service}/types` not `capabilities/types` * decided not to unless requested (makes exports map even more complicated, can do later) * [x] GH actions should run tests on packages/capabilities * [x] .github/workflows/release.yml does npm/docs jobs if paths_released contains packages/capabilities Co-authored-by: Irakli Gozalishvili <[email protected]>
Motivation: * #80 (comment) * trying to get release-please-action to generate a release PR for packages/capabilities * I would have expected it in this action, but there was [an error](https://github.com/web3-storage/w3protocol/actions/runs/3587365143/jobs/6037622712#step:2:44) ``` Expected 5 releases, only found 4 ⚠ Missing 1 paths: packages/capabilities ⚠ No version for path packages/capabilities ``` Theory: * maybe these errors are because I hadn't added this package to the manifest?
…check for conventional-commits names in PRs (#233) Motivation: * This didn't trigger a release-please #232 * Investigate: * hmmmm "✔ No user facing commits found since beginning of time - skipping" https://github.com/web3-storage/w3protocol/actions/runs/3587365143/jobs/6037622712#step:2:368 * Act * this PR adds a commit inside packages/capabilities
🤖 I have created a release *beep* *boop* --- ## 1.0.0 (2022-12-01) ### Bug Fixes * add exports to packages/capabilities/readme.md, add workflow to check for conventional-commits names in PRs ([#233](#233)) ([c567d25](c567d25)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Motivation: * part of #237 * we need this version of ucanto libs because they include updated types to handle non-key-dids and dids with more than one key Parts * [x] build/test works in packages/access-client * [x] build/test works in packages/upload-client * [x] [blocked by needed ucanto release](storacha/ucanto#163) * [x] build/test works in packages/access-api * [x] [blocked by needed ucanto release](storacha/ucanto#163) Co-authored-by: Irakli Gozalishvili <[email protected]>
…0" (#245) Reverts #238 (requested by @hugomrdias )
- update tests and types to handle upload/list items with a shards array property. - update upload/add fn to return an UploadAddResponse as shards out may be more than shards in. - update README see also: storacha/w3infra#56 Signed-off-by: Oli Evans <[email protected]>
This will be the keyring
access-client cli should support #153 changes - [x] improve d1 errors .ie space already registered - [x] tests to validate that register space saves all the delegations and updates isRegistered - [x] setup cmd - [x] whoami cmd - [x] create space cmd - [x] space info - [x] delegate caps - [x] import space from delegation - [x] recover with client - [x] tests migrations actually handles it properly and keeps track of migration already applied - [x] d1 spaces table now stores metadata and the invocation that registered the space - [x] we need some names on the delegations recovered so the user can know what they are, maybe put it in the facts ? - [x] remove duplication on the on `waitForSpaceRecover` and `waitForVoucherRedeem`
Motivation: * latest commit on main branch was failing release workflow [here](https://github.com/web3-storage/w3protocol/actions/runs/3621669639/jobs/6105393187#step:2:35) * I think it's because the release-please config/manifest still referred to `package/wallet`, which this PR removes * seems like logical followup to #248 Co-authored-by: Hugo Dias <[email protected]>
superseedes #139 <img width="897" alt="Screenshot 2022-12-06 at 16 47 14" src="https://user-images.githubusercontent.com/314190/205972386-05fec7b8-e17b-4a3d-8f18-ede71e4b5ad1.png"> closes #139
Goals: 1. Decouple store from agent 2. Simplify agent creation 3. Agent governs data format not store 4. Initialization of agent data, not store 5. DRY initialization in agent, not repeated in each store impl ~~See: https://gist.github.com/alanshaw/ea4bd2b0ab215ade696eac1300be577d~~ outdated ```js // In regular use: const store = new StoreIndexedDB() const data = await store.load() const agent = data ? Agent.from(data, { store }) : await Agent.create({}, { store }) // Then StoreMemory can be deleted, since the AgentData already stores all data in // memory. It's equivalent to: const agent = await Agent.create() ```
🤖 I have created a release *beep* *boop* --- ## [8.0.0](access-v7.0.2...access-v8.0.0) (2022-12-06) ### ⚠ BREAKING CHANGES * access-client store decoupling ([#228](#228)) * upgrade to `@ucanto/{interface,principal}`@^4.0.0 ([#238](#238)) * follow up on the capabilities extract ([#239](#239)) ### Features * **access-client:** cli and recover ([#207](#207)) ([720dafb](720dafb)) * follow up on the capabilities extract ([#239](#239)) ([717fcaa](717fcaa)) * Revert "feat!: upgrade to `@ucanto/{interface,principal}`@^4.0.0" ([#245](#245)) ([197439e](197439e)) * upgrade to `@ucanto/{interface,principal}`@^4.0.0 ([#238](#238)) ([309aff0](309aff0)) ### Bug Fixes * connection method is not async ([#222](#222)) ([7c4e0b6](7c4e0b6)) ### Code Refactoring * access-client store decoupling ([#228](#228)) ([64c7496](64c7496)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop* --- ## [3.0.0](upload-client-v2.1.0...upload-client-v3.0.0) (2022-12-06) ### ⚠ BREAKING CHANGES * upload/list items have a shards array property ([#229](#229)) * upgrade to `@ucanto/{interface,principal}`@^4.0.0 ([#238](#238)) * follow up on the capabilities extract ([#239](#239)) ### Features * **access-client:** cli and recover ([#207](#207)) ([720dafb](720dafb)) * follow up on the capabilities extract ([#239](#239)) ([717fcaa](717fcaa)) * Revert "feat!: upgrade to `@ucanto/{interface,principal}`@^4.0.0" ([#245](#245)) ([197439e](197439e)) * upgrade to `@ucanto/{interface,principal}`@^4.0.0 ([#238](#238)) ([309aff0](309aff0)) * upload/list items have a shards array property ([#229](#229)) ([723b281](723b281)) ### Bug Fixes * list response results return type ([#206](#206)) ([e49c685](e49c685)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop* --- ## [8.0.1](access-v8.0.0...access-v8.0.1) (2022-12-07) ### Bug Fixes * conf driver can store top level undefined ([5abc522](5abc522)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
…#246) Motivation: * second attempt at this #238 * after first was reverted #245 Warning: * the versions of ucanto we're upgrading to were backward-incompatible. storacha/ucanto#149 * Old 'SignerArchive' exports will probably no longer import using the new `@ucanto/principal@^4.0.0` `SignerArchive` * in slack, @alanshaw said he thinks this is unlikely to cause problems: "There’s no code in production that uses toArchive" Blockers * [x] Wait for these two to land first (@hugomrdias says so) * [x] @hugomrdias #207 * [x] @alanshaw #228 * [x] then will need to resolve merge conflicts on this branch * [x] @gobengo resolve conflicts after #207 * [x] @gobengo resolve conflicts after #228 Co-authored-by: Irakli Gozalishvili <[email protected]>
🤖 I have created a release *beep* *boop* --- ## [9.0.0](access-v8.0.1...access-v9.0.0) (2022-12-07) ### ⚠ BREAKING CHANGES * upgrade access-api @ucanto/* and @ipld/dag-ucan major versions ([#246](#246)) ### Features * upgrade access-api @ucanto/* and @ipld/dag-ucan major versions ([#246](#246)) ([65d191c](65d191c)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop* --- ## [4.0.0](upload-client-v3.0.0...upload-client-v4.0.0) (2022-12-07) ### ⚠ BREAKING CHANGES * upgrade access-api @ucanto/* and @ipld/dag-ucan major versions ([#246](#246)) ### Features * upgrade access-api @ucanto/* and @ipld/dag-ucan major versions ([#246](#246)) ([65d191c](65d191c)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop* --- ## [5.0.0](upload-client-v4.0.0...upload-client-v5.0.0) (2022-12-07) ### ⚠ BREAKING CHANGES * upgrade access-api @ucanto/* and @ipld/dag-ucan major versions ([#246](#246)) * upload/list items have a shards array property ([#229](#229)) * upgrade to `@ucanto/{interface,principal}`@^4.0.0 ([#238](#238)) * follow up on the capabilities extract ([#239](#239)) * rename callback ([#198](#198)) ### Features * [#153](#153) ([#177](#177)) ([7fbf2a1](7fbf2a1)) * a base for a new web3.storage upload-client ([#141](#141)) ([72c08d7](72c08d7)) * **access-client:** cli and recover ([#207](#207)) ([720dafb](720dafb)) * account recover with email ([#149](#149)) ([91ad47d](91ad47d)) * add static uploads API ([#148](#148)) ([8b735fa](8b735fa)) * allow custom shard size ([#185](#185)) ([2130405](2130405)) * explicit resource ([#181](#181)) ([c127431](c127431)) * follow up on the capabilities extract ([#239](#239)) ([717fcaa](717fcaa)) * Revert "feat!: upgrade to `@ucanto/{interface,principal}`@^4.0.0" ([#245](#245)) ([197439e](197439e)) * support pagination ([#204](#204)) ([363e3ac](363e3ac)), closes [#201](#201) * upgrade access-api @ucanto/* and @ipld/dag-ucan major versions ([#246](#246)) ([65d191c](65d191c)) * upgrade to `@ucanto/{interface,principal}`@^4.0.0 ([#238](#238)) ([309aff0](309aff0)) * upload/list items have a shards array property ([#229](#229)) ([723b281](723b281)) ### Bug Fixes * export types ([c44a252](c44a252)) * list response results return type ([#206](#206)) ([e49c685](e49c685)) * store export ([#197](#197)) ([d2296c5](d2296c5)) * upload client list needs empty nb ([#194](#194)) ([7894fb9](7894fb9)) * uploadFile requires BlobLike not Blob ([56ec496](56ec496)) ### Code Refactoring * rename callback ([#198](#198)) ([2349763](2349763)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop* --- ## [2.0.0](capabilities-v1.0.0...capabilities-v2.0.0) (2022-12-07) ### ⚠ BREAKING CHANGES * upgrade access-api @ucanto/* and @ipld/dag-ucan major versions ([#246](#246)) * upgrade to `@ucanto/{interface,principal}`@^4.0.0 ([#238](#238)) * follow up on the capabilities extract ([#239](#239)) ### Features * **access-client:** cli and recover ([#207](#207)) ([720dafb](720dafb)) * follow up on the capabilities extract ([#239](#239)) ([717fcaa](717fcaa)) * Revert "feat!: upgrade to `@ucanto/{interface,principal}`@^4.0.0" ([#245](#245)) ([197439e](197439e)) * upgrade access-api @ucanto/* and @ipld/dag-ucan major versions ([#246](#246)) ([65d191c](65d191c)) * upgrade to `@ucanto/{interface,principal}`@^4.0.0 ([#238](#238)) ([309aff0](309aff0)) ### Bug Fixes * fix Access API cannot get space/info [#243](#243) ([#255](#255)) ([1a74031](1a74031)) * generated typdefs ([#258](#258)) ([dfd2bd5](dfd2bd5)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: Benjamin Goering <[email protected]>
Cannot put `undefined` in UCANs
🤖 I have created a release *beep* *boop* --- ## [9.0.1](access-v9.0.0...access-v9.0.1) (2022-12-08) ### Bug Fixes * validate agent name ([#271](#271)) ([5a1fdb2](5a1fdb2)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
I noticed earlier today that running `pnpm run docs` fails with a bunch of typescript errors unless you run `pnpm run build` first. This just updates the `docs` npm script to `pnpm run build && typedoc --out docs`, so we'll always have build output to make typedoc happy.
Buffer has a toJSON implementation which is called before our replacer
function when serialising. It's JSON form is an object like `{ type:
'Buffer, data: number[] }`.
This means it doesn't get identfied as $bytes, and does not get
deserialised in the reviver. I think we'd rather have `Uint8Array`
everywhere so this PR fixes it so that it gets revived as that.
License: MIT
Signed-off-by: Oli Evans <[email protected]>
make it consistent with other packages in this monorepo so that the top level prettier format command doesn't make unexpected changes when trying to format other packages
This is web3-storage/w3up-client#74 ported here. --- This PR allows passing your own principal to the client `create` factory function instead of loading one from a pre-existing store. The caveat is that if you pass your own and the store is not empty then it needs to match the principal that was loaded from the store (otherwise any saved delegations will be unusable). This makes https://gist.github.com/alanshaw/e949abfcf6728f590ac9fa083dba5648#on-the-server a little easier, you won't need to install `@web3-storage/access` or deal with the `AgentData` class. Currently: ```js import * as Signer from '@ucanto/principal/ed25519' import { AgentData } from '@web3-storage/access/agent' import { Client } from '@web3-storage/w3up-client' const principal = Signer.parse(process.env.KEY) const data = await AgentData.create({ principal }) const client = new Client(data) ``` After this PR: ```js import * as Signer from '@ucanto/principal/ed25519' import * as Client from '@web3-storage/w3up-client' const principal = Signer.parse(process.env.KEY) const client = await Client.create({ principal }) ```
🤖 I have created a release *beep* *boop* --- ## [5.4.0](w3up-client-v5.3.0...w3up-client-v5.4.0) (2023-03-29) ### Features * bring your own principal ([#672](#672)) ([afade7c](afade7c)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
…ve some type checking (#669)
Previously `this.proofs(caps)` was returning `[]` when you request capabilities the agent is not authorized to use, the call to `delegate` would succeed and you get a useless delegation because there are no proofs.
Build on @Bengo's `access/claim`-polling-based `waitForDelegationOnSocket` replacement. This will let us replace the websocket-based authorization wait function which makes the UX much more responsive - rather than waiting an indeterminate amount of time after clicking the email link, the process completes almost immediately with this change. The wait function is also pluggable so we that clients can implement different polling strategies, which should make it easier to test and use a receipt-based version of this in the future. Added tests for this that use `sinon` to mock out service handlers and ensure they are called the right number of times.
🤖 I have created a release *beep* *boop* --- ## [11.1.0](access-v11.0.1...access-v11.1.0) (2023-03-29) ### Features * get `access/claim` authorization wait function working ([#666](#666)) ([1d0e208](1d0e208)) ### Bug Fixes * verify proofs exist for requested delegation capabilities ([#670](#670)) ([a0cea8f](a0cea8f)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop* --- ## [8.2.0](upload-client-v8.1.0...upload-client-v8.2.0) (2023-03-29) ### Features * add w3up-client at /packages/w3up-client ([#653](#653)) ([7129181](7129181)) ### Bug Fixes * catch p-queue error on abort ([e4b2ce8](e4b2ce8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: Alan Shaw <[email protected]>
🤖 I have created a release *beep* *boop* --- ## [5.5.0](w3up-client-v5.4.0...w3up-client-v5.5.0) (2023-03-29) ### Features * add capabilities option type for authorize ([#687](#687)) ([57697ad](57697ad)) * get `access/claim` authorization wait function working ([#666](#666)) ([1d0e208](1d0e208)) ### Bug Fixes * missing file from byo principal ([ac9b413](ac9b413)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Removes the restriction that importing a space should have `*` capability. Alternative to #671
🤖 I have created a release *beep* *boop* --- ## [11.2.0](access-v11.1.0...access-v11.2.0) (2023-03-29) ### Features * allow importing a space with restricted abilities ([#685](#685)) ([8996121](8996121)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Motivation: * try to fix #694
Motivation: * dummy commit to test #694 * I expect when we merge this, I want to see release-please open a w3up-client release-please PR for 5.5.1
…d correctly (#719) … right, causing access/authorize not to send emails for email addresses with '+' Motivation: * storacha/w3cli#73
…722) Motivation: * #682 Breaking Changes * remove `createDidMailtoFromEmail` export from `@web3-storage/access/agent` 93e29c6#diff-69a4efe733b2d7920dc103a0370eb1a285403e39c41e1b5e9a0718ea66b5a32fL33 * no dependencies under our org: https://github.com/search?q=org%3Aweb3-storage%20createDidMailtoFromEmail&type=code * Motivation: encouragement [from @Gozala](#722 (comment)) and [@travis](#722 (comment)) in review
…dProgress callback (#720) Use the ipfs-utils fetch implementation to enable upload progress reporting. Users of `upload-client` can now pass an `onUploadProgress` callback to `uploadFile` and `uploadDirectory` which will be handed off to the `fetch` polyfill from `ipfs-utils`. We wrap `onUploadProgress` in order to set `url` on the progress status events we get from `fetch` so that clients can differentiate between upload progress events from different shards, which will be interleaved.
🤖 I have created a release *beep* *boop* --- ## [8.3.0](upload-client-v8.2.0...upload-client-v8.3.0) (2023-04-05) ### Features * allow clients to pass uploadFile and uploadDirectory an onUploadProgress callback ([#720](#720)) ([20bc69f](20bc69f)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Gozala
force-pushed
the
feat/incremental-dir-upload
branch
from
abc1cea
to
6dd6b3c
Compare
gobengo
force-pushed
the
feat/incremental-dir-upload
branch
from
2565fee
to
9f61685
Compare
|
reopened as #742 because ThirdWeb is still using it |
|
Sorry to have accidentally pushed to this branch here and closed this. It was a mistake! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.