implement can store ls and can upload ls
#45
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
implement `can store ls` and `can upload ls` to parity with `ls`.
- fix spelling error - standardize on * for arg lists in readme - add docs for `can upload ls` and `can store ls`
add support for setting size and cursor caveats in `can store ls` and `can upload ls` to support this usage, also add a `--raw` output formatting option to `ls`, `can store ls` and `can upload ls` that will allow users to get paging cursors from the response
this allows `can store ls` and `can upload ls` to return the results preceding the given cursor.
I had broken json formatting and raw formatting never actually worked
I added a bunch of new options to the list commands - document them
also improve JSDocs for new lib.js functions I'm not entirely sure the bin test mocks are done properly - would love careful review there
olizilla
reviewed
Feb 14, 2023
olizilla
reviewed
Feb 14, 2023
travis
added a commit
to storacha/w3infra
that referenced
this pull request
Feb 16, 2023
Implement support for the `pre` caveat introduced in storacha/w3up#423 Also introduce `startCursor` and `endCursor` with the intention to deprecate `cursor`. By providing cursors for the beginning and end of the range, we enable clients to page forward or backward without saving state anywhere. Example usage: Given uploads like: ```bash $ w3 ls bafybeia7tr4dgyln7zeyyyzmkppkcts6azdssykuluwzmmswysieyadcbm bafybeibvbxjeodaa6hdqlgbwmv4qzdp3bxnwdoukay4dpl7aemkiwc2eje bafybeid2fbwlsb7euczxw7ig3q3fyezw7tfkcikrer75w4ucfawh5vqfdm bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji bafybeiefiddap3p3z3p3obatviipgda2us3umt4uxeh2trrni5pt6bfc6e bafybeifohbynkellbberk3hi447dntungwgicmsvxuwdttk5k5k535mw2a bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq bafybeign7hb7ynhc6o3sevas365xuzuqzrhpfzgksi4dzpcbphl4xrerye bafybeignyyeaqn6b2fzqswrozt2txexjbwesrt4dbxlquoac4nn7mbi2gm bafybeigz7o2ibtv6k5pnklfza3ebsl6jeveguoyxwvrmmtvp2lkvqqjyem ``` You can now page forward like: ```bash $ w3 can upload ls --size 4 bafybeia7tr4dgyln7zeyyyzmkppkcts6azdssykuluwzmmswysieyadcbm bafybeibvbxjeodaa6hdqlgbwmv4qzdp3bxnwdoukay4dpl7aemkiwc2eje bafybeid2fbwlsb7euczxw7ig3q3fyezw7tfkcikrer75w4ucfawh5vqfdm bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy ``` ```bash $ w3 can upload ls --size 4 --cursor bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji bafybeiefiddap3p3z3p3obatviipgda2us3umt4uxeh2trrni5pt6bfc6e bafybeifohbynkellbberk3hi447dntungwgicmsvxuwdttk5k5k535mw2a bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq ``` ```bash $ w3 can upload ls --size 4 --cursor bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq bafybeign7hb7ynhc6o3sevas365xuzuqzrhpfzgksi4dzpcbphl4xrerye bafybeignyyeaqn6b2fzqswrozt2txexjbwesrt4dbxlquoac4nn7mbi2gm bafybeigz7o2ibtv6k5pnklfza3ebsl6jeveguoyxwvrmmtvp2lkvqqjyem ``` and page backward like: ```bash $ w3 can upload ls --size 4 --cursor bafybeign7hb7ynhc6o3sevas365xuzuqzrhpfzgksi4dzpcbphl4xrerye --pre bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq bafybeifohbynkellbberk3hi447dntungwgicmsvxuwdttk5k5k535mw2a bafybeiefiddap3p3z3p3obatviipgda2us3umt4uxeh2trrni5pt6bfc6e bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji ``` ```bash $ w3 can upload ls --size 4 --cursor bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji --pre bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy bafybeid2fbwlsb7euczxw7ig3q3fyezw7tfkcikrer75w4ucfawh5vqfdm bafybeibvbxjeodaa6hdqlgbwmv4qzdp3bxnwdoukay4dpl7aemkiwc2eje bafybeia7tr4dgyln7zeyyyzmkppkcts6azdssykuluwzmmswysieyadcbm ``` Note that the cursor values here happen to match the CID strings - this is an implementation detail and should not be relied upon. Instead, cursor values should be taken from `startCursor` and `endCursor` in the list response: ```bash $ w3 can upload ls --size 4 --cursor bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy --raw {"size":4,"cursor":"bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq","results":[{"root":{"/":"bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji"},"shards":[{"/":"bagbaieracmkgwrw6rowsk5jse5eihyhszyrq5w23aqosajyckn2tfbotdcqq"}],"updatedAt":"2023-02-13T16:30:04.172Z","insertedAt":"2023-02-13T16:30:04.172Z"},{"root":{"/":"bafybeiefiddap3p3z3p3obatviipgda2us3umt4uxeh2trrni5pt6bfc6e"},"shards":[{"/":"bagbaieraz3iyicghnyzpjld7yxf7rmntf5cdbewo4pmuck6kkkc6f7z3sznq"}],"updatedAt":"2023-02-13T16:29:51.864Z","insertedAt":"2023-02-13T10:57:39.017Z"},{"root":{"/":"bafybeifohbynkellbberk3hi447dntungwgicmsvxuwdttk5k5k535mw2a"},"shards":[{"/":"bagbaierasu5cufqagk34dkmhszbwuhiydjjjp3pl4bacyvskljqtkcxhgbta"}],"updatedAt":"2023-02-13T16:30:11.648Z","insertedAt":"2023-02-13T10:57:16.264Z"},{"root":{"/":"bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq"},"shards":[{"/":"bagbaieralqhbrq2mqupqijkoylagbams263jwq6helsl6kbdpivupjm7egxq"}],"updatedAt":"2023-02-13T16:30:18.403Z","insertedAt":"2023-02-13T10:57:49.695Z"}], "endCursor":"bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq","startCursor":"bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji"} ``` nb that these w3 commands require the changes I implemented in storacha/w3cli#45 --------- Co-authored-by: Benjamin Goering <[email protected]>
alanshaw
requested changes
Feb 20, 2023
because we automatically page through all results in this command, it doesn't make sense to print the raw response. users interested individual page-level responses can use `can upload ls`
Per the change in storacha/w3infra#148
|
requested changes should be implemented - should be good to go for a re-review! |
alanshaw
requested changes
Feb 24, 2023
There was a problem hiding this comment.
Ok, I've been thinking a bit about this...there's a couple of things I'm not 100% sold on:
- Adding another flag (
--raw) that does almost the same thing as--json - Making a human user paginate manually, like an animal
The proposal:
- Pull
--rawfrom this PR and open an issue for exposing the cursor - Open an issue for listing items for humans - this might be a "Continue (y/n)?" prompt or a
--streamflag (or both)
as requested by @alanshaw in #45 (review), back out --raw for now
This was referenced Mar 1, 2023
|
ok @alanshaw I think this should be ready to merge |
alanshaw
approved these changes
Mar 1, 2023
lib.js
Outdated
| @@ -13,6 +14,9 @@ import { CarReader } from '@ipld/car' | |||
|
|
|||
| /** | |||
| * @typedef {import('@web3-storage/w3up-client/types').FileLike & { size: number }} FileLike | |||
| * @typedef {import('@web3-storage/w3up-client/src/types').ListResponse} ListResponse | |||
| * @typedef {import('@web3-storage/w3up-client/src/types').StoreListResult} StoreListResult | |||
| * @typedef {import('@web3-storage/w3up-client/src/types').UploadListResult} UploadListResult | |||
There was a problem hiding this comment.
Why @web3-storage/w3up-client/src/types not @web3-storage/w3up-client/types?
There was a problem hiding this comment.
oh good question, I suspect I copy/pasted this from somewhere (funny because the path I was looking for was right there in the previous line) - will fix
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement
can store lsandcan upload lscommands, including support for theprecaveat introduced in storacha/w3up#423 and support for thesizeandcursorcaveats, set with options of the same name.Also introduce a
--rawoption to print out the raw JSON response fromlscommands.Also make some docs changes for more consistency and less speling erors.
Needs a new version of w3up-client once web3-storage/w3up-client#77 is merged and released.