Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement can store ls and can upload ls #45

Merged
merged 17 commits into from
Mar 2, 2023
Merged

implement can store ls and can upload ls #45

merged 17 commits into from
Mar 2, 2023

Conversation

travis
Copy link
Member

@travis travis commented Feb 13, 2023

Implement can store ls and can upload ls commands, including support for the pre caveat introduced in storacha/w3up#423 and support for the size and cursor caveats, set with options of the same name.

Also introduce a --raw option to print out the raw JSON response from ls commands.

Also make some docs changes for more consistency and less speling erors.

Needs a new version of w3up-client once web3-storage/w3up-client#77 is merged and released.

implement `can store ls` and `can upload ls` to parity with `ls`.
- fix spelling error
- standardize on * for arg lists in readme
- add docs for `can upload ls` and `can store ls`
add support for setting size and cursor caveats in `can store ls` and `can upload ls`

to support this usage, also add a `--raw` output formatting option to `ls`, `can store ls` and `can upload ls` that will allow users to get paging cursors from the response
this allows `can store ls` and `can upload ls` to return the results
preceding the given cursor.
I had broken json formatting and raw formatting never actually worked
I added a bunch of new options to the list commands - document them
also improve JSDocs for new lib.js functions

I'm not entirely sure the bin test mocks are done properly - would love careful review there
README.md Outdated Show resolved Hide resolved
can.js Show resolved Hide resolved
travis added a commit to storacha/w3infra that referenced this pull request Feb 16, 2023
Implement support for the `pre` caveat introduced in
storacha/w3up#423

Also introduce `startCursor` and `endCursor` with the intention to
deprecate `cursor`. By providing cursors for the beginning and end of
the range, we enable clients to page forward or backward without saving
state anywhere.

Example usage:

Given uploads like:

```bash
$ w3 ls
bafybeia7tr4dgyln7zeyyyzmkppkcts6azdssykuluwzmmswysieyadcbm
bafybeibvbxjeodaa6hdqlgbwmv4qzdp3bxnwdoukay4dpl7aemkiwc2eje
bafybeid2fbwlsb7euczxw7ig3q3fyezw7tfkcikrer75w4ucfawh5vqfdm
bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy
bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji
bafybeiefiddap3p3z3p3obatviipgda2us3umt4uxeh2trrni5pt6bfc6e
bafybeifohbynkellbberk3hi447dntungwgicmsvxuwdttk5k5k535mw2a
bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq
bafybeign7hb7ynhc6o3sevas365xuzuqzrhpfzgksi4dzpcbphl4xrerye
bafybeignyyeaqn6b2fzqswrozt2txexjbwesrt4dbxlquoac4nn7mbi2gm
bafybeigz7o2ibtv6k5pnklfza3ebsl6jeveguoyxwvrmmtvp2lkvqqjyem
```

You can now page forward like:

```bash
$ w3 can upload ls --size 4
bafybeia7tr4dgyln7zeyyyzmkppkcts6azdssykuluwzmmswysieyadcbm
bafybeibvbxjeodaa6hdqlgbwmv4qzdp3bxnwdoukay4dpl7aemkiwc2eje
bafybeid2fbwlsb7euczxw7ig3q3fyezw7tfkcikrer75w4ucfawh5vqfdm
bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy
```

```bash
$ w3 can upload ls --size 4 --cursor bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy
bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji
bafybeiefiddap3p3z3p3obatviipgda2us3umt4uxeh2trrni5pt6bfc6e
bafybeifohbynkellbberk3hi447dntungwgicmsvxuwdttk5k5k535mw2a
bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq
```

```bash
$ w3 can upload ls --size 4 --cursor bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq
bafybeign7hb7ynhc6o3sevas365xuzuqzrhpfzgksi4dzpcbphl4xrerye
bafybeignyyeaqn6b2fzqswrozt2txexjbwesrt4dbxlquoac4nn7mbi2gm
bafybeigz7o2ibtv6k5pnklfza3ebsl6jeveguoyxwvrmmtvp2lkvqqjyem
```

and page backward like:

```bash
$ w3 can upload ls --size 4 --cursor bafybeign7hb7ynhc6o3sevas365xuzuqzrhpfzgksi4dzpcbphl4xrerye --pre
bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq
bafybeifohbynkellbberk3hi447dntungwgicmsvxuwdttk5k5k535mw2a
bafybeiefiddap3p3z3p3obatviipgda2us3umt4uxeh2trrni5pt6bfc6e
bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji
```

```bash
$ w3 can upload ls --size 4 --cursor bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji --pre
bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy
bafybeid2fbwlsb7euczxw7ig3q3fyezw7tfkcikrer75w4ucfawh5vqfdm
bafybeibvbxjeodaa6hdqlgbwmv4qzdp3bxnwdoukay4dpl7aemkiwc2eje
bafybeia7tr4dgyln7zeyyyzmkppkcts6azdssykuluwzmmswysieyadcbm
```

Note that the cursor values here happen to match the CID strings - this
is an implementation detail and should not be relied upon. Instead,
cursor values should be taken from `startCursor` and `endCursor` in the
list response:

```bash
$ w3 can upload ls --size 4 --cursor bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy --raw
{"size":4,"cursor":"bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq","results":[{"root":{"/":"bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji"},"shards":[{"/":"bagbaieracmkgwrw6rowsk5jse5eihyhszyrq5w23aqosajyckn2tfbotdcqq"}],"updatedAt":"2023-02-13T16:30:04.172Z","insertedAt":"2023-02-13T16:30:04.172Z"},{"root":{"/":"bafybeiefiddap3p3z3p3obatviipgda2us3umt4uxeh2trrni5pt6bfc6e"},"shards":[{"/":"bagbaieraz3iyicghnyzpjld7yxf7rmntf5cdbewo4pmuck6kkkc6f7z3sznq"}],"updatedAt":"2023-02-13T16:29:51.864Z","insertedAt":"2023-02-13T10:57:39.017Z"},{"root":{"/":"bafybeifohbynkellbberk3hi447dntungwgicmsvxuwdttk5k5k535mw2a"},"shards":[{"/":"bagbaierasu5cufqagk34dkmhszbwuhiydjjjp3pl4bacyvskljqtkcxhgbta"}],"updatedAt":"2023-02-13T16:30:11.648Z","insertedAt":"2023-02-13T10:57:16.264Z"},{"root":{"/":"bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq"},"shards":[{"/":"bagbaieralqhbrq2mqupqijkoylagbams263jwq6helsl6kbdpivupjm7egxq"}],"updatedAt":"2023-02-13T16:30:18.403Z","insertedAt":"2023-02-13T10:57:49.695Z"}],
 "endCursor":"bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq","startCursor":"bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji"}
```

nb that these w3 commands require the changes I implemented in
storacha/w3cli#45

---------

Co-authored-by: Benjamin Goering <[email protected]>
@travis travis requested a review from olizilla February 17, 2023 08:17
index.js Show resolved Hide resolved
README.md Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
because we automatically page through all results in this command, it doesn't make sense to print the raw response. users interested individual page-level responses can use `can upload ls`
@travis travis requested a review from alanshaw February 21, 2023 06:27
@travis
Copy link
Member Author

travis commented Feb 24, 2023

requested changes should be implemented - should be good to go for a re-review!

Copy link
Member

@alanshaw alanshaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've been thinking a bit about this...there's a couple of things I'm not 100% sold on:

  1. Adding another flag (--raw) that does almost the same thing as --json
  2. Making a human user paginate manually, like an animal

The proposal:

  • Pull --raw from this PR and open an issue for exposing the cursor
  • Open an issue for listing items for humans - this might be a "Continue (y/n)?" prompt or a --stream flag (or both)

index.js Show resolved Hide resolved
test/bin.spec.js Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
@travis
Copy link
Member Author

travis commented Mar 1, 2023

  • Pull --raw from this PR and open an issue for exposing the cursor
  • Open an issue for listing items for humans - this might be a "Continue (y/n)?" prompt or a --stream flag (or both)

ok done in #47 and #49

@travis travis requested a review from alanshaw March 1, 2023 05:12
@travis
Copy link
Member Author

travis commented Mar 1, 2023

ok @alanshaw I think this should be ready to merge

Copy link
Member

@alanshaw alanshaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

lib.js Outdated
@@ -13,6 +14,9 @@ import { CarReader } from '@ipld/car'

/**
* @typedef {import('@web3-storage/w3up-client/types').FileLike & { size: number }} FileLike
* @typedef {import('@web3-storage/w3up-client/src/types').ListResponse} ListResponse
* @typedef {import('@web3-storage/w3up-client/src/types').StoreListResult} StoreListResult
* @typedef {import('@web3-storage/w3up-client/src/types').UploadListResult} UploadListResult
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why @web3-storage/w3up-client/src/types not @web3-storage/w3up-client/types?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh good question, I suspect I copy/pasted this from somewhere (funny because the path I was looking for was right there in the previous line) - will fix

@travis travis merged commit 4ad8f4c into main Mar 2, 2023
@travis travis deleted the feat/can-ls branch March 2, 2023 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants