QueryBuilder: Implementation of contains Filter Operator applied to JSON Fields for SQLite backend
#6619
Conversation
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
rabbull
changed the title
contains Filter Operator applied to JSON Fields for SQLite backendcontains Filter Operator applied to JSON Fields for SQLite backend
for more information, see https://pre-commit.ci
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6619 +/- ##
==========================================
+ Coverage 78.00% 78.06% +0.07%
==========================================
Files 563 563
Lines 41766 41790 +24
==========================================
+ Hits 32574 32618 +44
+ Misses 9192 9172 -20 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Let's maybe shelve this for now, until we have resolved contains for PSQL? Also, just out of curiosity, any reason for going through the custom implementation, rather than through SQLA, apart from the weird behavior for PSQL?
If we agree that the behavior for PSQL is fine, I'd vouch that we also go the SQLA route here. We just need to make sure the behavior is consistent for both backends, as well as document it somewhere.
The main reason for opting for a custom implementation over SQLA is it's too hard to implement such an operator in SQL, as well as in SQLA, due to the insufficient support of recursive operations and effective branching in SQL. Even if a viable solution could be carried out, it is most likely to be unreadable and unmaintainable. |
…into sqlite-json-contains
for more information, see https://pre-commit.ci
|
Hi @GeigerJ2, I’ve updated the tests for this feature by combining all tests for JSON filters dedicated on the SQLite backend into those on the PostgreSQL backend, and removed As for the placement of the custom function Please kindly review the new modification when you are available. |
rabbull
changed the title
contains Filter Operator applied to JSON Fields for SQLite backendcontains Filter Operator applied to JSON Fields for SQLite backend
for more information, see https://pre-commit.ci
|
Hi @rabbull, will review this PR soon. Sorry for the delay! |
No worries. Take your time. I was just trying to figure out why it is blocked by the codecov/patch. It seems including files that I didn't changed. I assumed it was because my branch is out-of-date so that comparing to the main branch a lot files looks modified, but it seems not the case because it is still blocking even if I merged main branch :( |
rabbull
force-pushed
the
sqlite-json-contains
branch
from
322c3d6 to
fbb7ee3
Compare
|
Hooray! The CI has passed successfully. This PR is now ready for your review and merging. Thanks in advance! cc @GeigerJ2 |
|
Thanks for the great work, @rabbull! 🚀 |
…am#6619) This commit re-introduces the `contains` filter operator for the SQLite backend. The primary goal is to replicate the functionality of `contains` from PostgreSQL to achieve feature parity between the two different storage backends. Due to the particular (and sometimes unexpected) behavior of PostgreSQL's containment operator (aiidateam#6618), a pure SQL implementation would be highly complex and difficult to maintain. As the behavior of the `contains` operator should be _exactly_ the same, irrespective of which backend is being used, custom Python functions were chosen for the implementation (rather than SQLAlchemy's high-level abstractions). Comprehensive tests and benchmarks that cover all potential use cases for `contains` on the SQLite backend have been added.
This PR introduces the
containsfilter operator for the SQLite backend, implemented using custom functions rather than SQLAlchemy's high-level abstractions. Comprehensive tests have been included to cover all potential use cases forcontainson the SQLite backend, providing a detailed demonstration of its behavior across various scenarios.The primary goal of this PR is to replicate the functionality of the
containsoperator from PostgreSQL on SQLite. However, due to the complex and sometimes unexpected behavior of PostgreSQL's containment operator (#6618), a pure SQL implementation would be highly complex and difficult to maintain. To address this, custom functions were chosen, leveraging the flexibility of high-level programming languages (Python, in this case) to handle complex control flows more effectively.While invoking functions from other languages introduces potential performance limitations, this trade-off is acceptable compared to the alternative of lacking this functionality entirely. Benchmarks will be added to assess the performance impact and ensure it remains within acceptable bounds.
This implementation serves as a proof of concept to evaluate the feasibility of using custom functions for the
containsoperator. While functional, the current code structure is still preliminary and requires significant cleanup. If this approach proves successful, a subsequent refactor will address technical debt and improve the maintainability of the codebase. This PR is marked as WIP while the approach is evaluated, benchmarks are added, and the necessary refactor is performed.Closes #6494