-
Notifications
You must be signed in to change notification settings - Fork 135
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(client): Fix batch deletes causing call stack overflow #1394
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left some comments, mainly simplifications for the unit tests.
Also don't forget to add a changeset.
clients/typescript/src/migrators/query-builder/sqliteBuilder.ts
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
small nitpicks, but GJ otherwise!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great job, left a few minor comments.
a6cfa1d
to
ad2e578
Compare
Addresses VAX-1985
Apparently the method we were using, where we chained
OR
andAND
clauses in theWHERE
clause, led to:wa-sqlite
) at around 6kOR
clauses, far below the max parameter limitI've converted the batching to use
IN
statements for both single-column queries and multi-column ones, like with composite primary keys. The performance is significantly improved and has room for more improvement:Old Batching
Shadow Table 30k deletes
Comment 9k deletes
New Batching
Shadow Table 30k deletes
Comment 9k deletes
While the shadow table deletes take similar time for wa-sqlite (~500ms), the shadow table uses a triple composite key of 3 string columns. There's room for significant optimization there, concatenating the parameters and deleting based on the concatenated PK columns reduced the time to execute from ~500ms to ~50ms - pglite also had a very small improvement but hard to measure.
I think it's possible to avoid having a composite primary key for the shadow table since we know it's 3 string columns and they can be collapsed into one if that proves to be a significant optimization, but that is outside of the scope of this fix, just raising it here as a potential optimization.
NOTE: this improvement is the result of the same investigation that yielded #1389, both together result in a useable linearlite with >2k issues otherwise we run into timeouts and call stack overflows.