Add support for basic cursors and limits to LookupSubjects by josephschorr · Pull Request #1379 · authzed/spicedb

josephschorr · 2023-06-01T21:33:15Z

This change supports a limit (called the "concrete limit") on LookupSubjects and will filter concrete subjects based on the returned cursor.

This change does not filter intermediate lookups, which will be done in a followup PR.

vroldanbet

I didn't have time to finish the review, sorry! This is my first attempt. So far it's looking good, although I have a bunch of questions.

Something I noticed is that when one resumes with a cursor, we are evaluating the whole graph again and sending queries to the database, even though they will return empty. It works, but is a lot of work that is wasted after resuming, and will be directly proportional to the complexity of the schema. Ideally the evaluation of the schema can also resume from where the cursor left off, e.g. if permission view = a + b + c you'd restart evaluation at c if that's where you left when the limit was hit. It would spare a bunch of dispatching (network calls across SpiceDBs!) and DB roundtrips.

vroldanbet · 2023-07-28T14:05:35Z

can we merge this into the loop below?

I did so, but I'm thinking maybe we just remove it entirely now? The field has been marked deprecated for a number of versions

sounds good to me!

vroldanbet · 2023-07-28T14:36:24Z

do we need to allocate a new encoded cursor each time here? Could we reuse the same proto instance and just modify the corresponding field? I did something as an optimization in the ReadRelationships streaming bits and helped a bunch with allocations and GC overhead. It would also spare us calling revision.String() repeatedly.

It would be far less maintainable, but I'll see what I can do

Done, but its a bit ugly

After a second pass, I thought the new code could have a race. Perhaps it's not worth the trouble? I'll leave it up to you

vroldanbet · 2023-07-28T17:18:56Z

in not sure to understand how we guarantee that we do not skip found subjects that are alphabetically after the current cursor but that haven't been seen by the stream before.

Because the stream always returns in sorted order from the root. That way, we are always guaranteed to have a defined ordering (alphabetical) coming out of each subproblem

Doesn't that mean that we have to start collecting subjects from all subproblems before we can start streaming? We can make sure that each subproblem retrieves results in order from DB, but you still need to execute all of them in order to determine what's the first subject to stream?

vroldanbet · 2023-08-21T19:53:06Z

there had been some regressions recently around dispatches and I wonder if we could use a new test-case that checks the number of dispatches over a cursored LR call

You mean over the LS call? We do now have the test for LR calls

I mean testing for LS. The logic should be different enough that it warrants another set of tests?

Gotcha. Added

vroldanbet · 2023-09-05T12:19:49Z

I see no use of the variadic allowedSubjectIds so please pass as a slice. NewSet should also support a slice - a lot of calls in the codebase that end up doing ... just to pass it to it.

Changed here but not NewSet. We have some places where the variadic is helpful. If you like, I can add another NewSetFromSlice and change the call sites?

yep I think that'd be a good idea

Done and changed all the call sites

josephschorr · 2023-09-06T17:57:58Z

Updated

josephschorr · 2024-03-11T17:41:04Z

Rebased

josephschorr · 2024-04-26T19:14:05Z

Rebased

vroldanbet

Doing a complete re-review to load all the context. First batch of feedback.

vroldanbet · 2024-05-16T12:06:08Z

The handling stream should handle all the elements up to the concrete limit, wouldn't it? From looking at the code, it would seem the only sole purpose is making sure the wildcard is sent, since in the worst-case scenario we stream up to the concrete limit, and we have to handle the potential +1 of the wildcard. Is that correct?

vroldanbet

@josephschorr last batch of feedback. Let's address this quickly so we don't loose the context and we can land it soon.

vroldanbet · 2024-05-17T13:59:50Z

I mean testing for LS. The logic should be different enough that it warrants another set of tests?

vroldanbet · 2024-05-17T14:02:18Z

After a second pass, I thought the new code could have a race. Perhaps it's not worth the trouble? I'll leave it up to you

vroldanbet · 2024-05-17T14:03:53Z

sounds good to me!

josephschorr · 2024-05-17T21:05:29Z

Updated

This change supports a limit (called the "concrete limit") on LookupSubjects and will filter concrete subjects based on the returned cursor. This change does *not* filter intermediate lookups, which will be done in a followup PR.

josephschorr · 2024-07-08T15:22:58Z

Moved to draft due to steelthread discovering a pagination bug

karlwang1983 · 2024-09-02T06:44:43Z

@vroldanbet Is there any progress for this pr, we are still looking forward the pagination function of LookupSubjects.

vroldanbet · 2024-09-02T09:20:43Z

@karlwang1983 I will defer to @josephschorr, my role has been to review the code.

mateenkasim · 2024-10-29T20:33:55Z

Also looking forward to this one!

josephschorr · 2025-01-06T18:17:11Z

Closing, as we will be taking a different approach to LookupSubjects pagination given that the current approach does not work as we intended

josephschorr requested review from a team and vroldanbet June 1, 2023 21:33

github-actions Bot added area/api v1 Affects the v1 API area/dependencies Affects dependencies area/dispatch Affects dispatching of requests area/tooling Affects the dev or user toolchain (e.g. tests, ci, build tools) labels Jun 1, 2023

josephschorr force-pushed the cursored-lookup-subjects branch 2 times, most recently from da48a6b to 19099a2 Compare June 1, 2023 21:50

josephschorr marked this pull request as draft June 20, 2023 23:38

josephschorr force-pushed the cursored-lookup-subjects branch 3 times, most recently from 21ae960 to c386a05 Compare July 12, 2023 19:36

josephschorr marked this pull request as ready for review July 12, 2023 19:58

vroldanbet reviewed Jul 28, 2023

View reviewed changes

vroldanbet reviewed Aug 21, 2023

View reviewed changes

vroldanbet reviewed Sep 5, 2023

View reviewed changes

Comment thread internal/dispatch/graph/lookupsubjects_test.go Outdated

Comment thread internal/graph/lookupsubjects.go Outdated

josephschorr force-pushed the cursored-lookup-subjects branch from c386a05 to 137e08e Compare September 6, 2023 17:57

github-actions Bot removed the area/dependencies Affects dependencies label Sep 6, 2023

josephschorr force-pushed the cursored-lookup-subjects branch from 137e08e to 1dec055 Compare March 11, 2024 17:40

josephschorr force-pushed the cursored-lookup-subjects branch 2 times, most recently from 3202a4b to b00913b Compare March 11, 2024 20:36

josephschorr force-pushed the cursored-lookup-subjects branch from b00913b to 83a96b2 Compare March 25, 2024 13:11

josephschorr force-pushed the cursored-lookup-subjects branch from 83a96b2 to 39f033f Compare April 26, 2024 19:13

josephschorr force-pushed the cursored-lookup-subjects branch 2 times, most recently from 57a79b9 to 07dbacd Compare May 14, 2024 15:48

vroldanbet reviewed May 16, 2024

View reviewed changes

vroldanbet reviewed May 17, 2024

View reviewed changes

josephschorr force-pushed the cursored-lookup-subjects branch from 07dbacd to 0c9fbb7 Compare May 17, 2024 21:05

vroldanbet mentioned this pull request May 20, 2024

introduce lookup subjects page limit authzed/zed#379

Closed

josephschorr force-pushed the cursored-lookup-subjects branch from 0c9fbb7 to 1f765d0 Compare May 20, 2024 18:33

github-actions Bot added the area/datastore Affects the storage system label May 20, 2024

josephschorr force-pushed the cursored-lookup-subjects branch from 1f765d0 to e74c6f3 Compare May 20, 2024 19:11

vroldanbet assigned josephschorr Jun 26, 2024

vroldanbet mentioned this pull request Jul 2, 2024

OptionalConcreteLimit and OptionalCursor on LookupSubjects doesn't work. #1964

Closed

josephschorr marked this pull request as draft July 8, 2024 15:22

josephschorr closed this Jan 6, 2025

github-actions Bot locked and limited conversation to collaborators Jan 6, 2025

Conversation

josephschorr commented Jun 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vroldanbet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vroldanbet Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vroldanbet May 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

josephschorr commented Sep 6, 2023

Uh oh!

josephschorr commented Mar 11, 2024

Uh oh!

josephschorr commented Apr 26, 2024

Uh oh!

vroldanbet left a comment

Choose a reason for hiding this comment

Uh oh!

josephschorr commented Jun 1, 2023 •

edited

Loading

vroldanbet Jul 28, 2023 •

edited

Loading

vroldanbet May 17, 2024 •

edited

Loading

vroldanbet May 17, 2024 •

edited

Loading