Fix PSQL OperationalError on archive creation at QB level #6998

GeigerJ2 · 2025-09-02T08:19:35Z

The most robust solution is (according to the internet and LLMs), a temporary table, and applying a join operation, rather than a massive SQL statement with many IN.

For future reference:

❯ pytest tests/storage/psql_dos/test_query.py::TestQueryBuilderParameterLimits::test_large_in_filter_batching --db-backend psql --pdb --pdbcls=IPython.terminal.debugger:Pdb -s
=================================================================================================== test session starts ===================================================================================================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core
configfile: pyproject.toml
plugins: cov-4.1.0, asyncio-0.16.0, timeout-2.3.1, regressions-2.5.0, rerunfailures-12.0, anyio-4.7.0, testmon-2.1.3, xdist-3.6.1, datadir-1.5.0, benchmark-4.0.0
timeout: 240.0s
timeout method: thread
timeout func_only: False
collected 1 item

tests/storage/psql_dos/test_query.py 09/02/2025 11:26:07 AM <135611> aiida.export: [REPORT] Parameters
-------------------------------  --------------------------------
Archive                          100k-int-nodes-2.7.1.post0.aiida
New Node Extras                  keep
Merge Node Extras (in database)  (k)eep
Merge Node Extras (in archive)   do (n)ot create
Merge Node Extras (in both)      (l)eave existing
Merge Comments                   leave
Computer Authinfos               exclude

09/02/2025 11:26:07 AM <135611> aiida.export: [REPORT] Adding 1 new user(s)
09/02/2025 11:26:07 AM <135611> aiida.export: [REPORT] Collecting Node(s) ...
09/02/2025 11:26:10 AM <135611> aiida.export: [REPORT] Adding 100000 new node(s)
09/02/2025 11:26:42 AM <135611> aiida.export: [REPORT] Created new import Group: PK=1, label=20250902-112642
09/02/2025 11:26:46 AM <135611> aiida.export: [REPORT] Checking keys against repository ...
09/02/2025 11:26:46 AM <135611> aiida.export: [REPORT] Committing transaction to database...
F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured log >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG    pgsu:__init__.py:147 Trying to connect via "psycopg"...
REPORT   aiida.export:log.py:34 Parameters
-------------------------------  --------------------------------
Archive                          100k-int-nodes-2.7.1.post0.aiida
New Node Extras                  keep
Merge Node Extras (in database)  (k)eep
Merge Node Extras (in archive)   do (n)ot create
Merge Node Extras (in both)      (l)eave existing
Merge Comments                   leave
Computer Authinfos               exclude

REPORT   aiida.export:log.py:34 Adding 1 new user(s)
REPORT   aiida.export:log.py:34 Collecting Node(s) ...
REPORT   aiida.export:log.py:34 Adding 100000 new node(s)
REPORT   aiida.export:log.py:34 Created new import Group: PK=1, label=20250902-112642
REPORT   aiida.export:log.py:34 Checking keys against repository ...
REPORT   aiida.export:log.py:34 Committing transaction to database...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

self = <sqlalchemy.engine.base.Connection object at 0x77d2bd587ee0>, dialect = <sqlalchemy.dialects.postgresql.psycopg.PGDialect_psycopg object at 0x77d2c4392e30>
context = <sqlalchemy.dialects.postgresql.psycopg.PGExecutionContext_psycopg object at 0x77d2bd585f30>, statement = <sqlalchemy.dialects.postgresql.psycopg.PGCompiler_psycopg object at 0x77d2bd585ff0>
parameters = [{'param_1': '%', 'uuid_10_1': '0a2d13ec-8826-41f0-960d-e77a86e2f272', 'uuid_10_10': '0de99d7d-88d4-4f2c-b9aa-005ed56ffa0c', 'uuid_10_100': '280bb0cf-f87b-4f5c-813c-c5c77d9b10fd', ...}]

    def _exec_single_context(
        self,
        dialect: Dialect,
        context: ExecutionContext,
        statement: Union[str, Compiled],
        parameters: Optional[_AnyMultiExecuteParams],
    ) -> CursorResult[Any]:
        """continue the _execute_context() method for a single DBAPI
        cursor.execute() or cursor.executemany() call.

        """
        if dialect.bind_typing is BindTyping.SETINPUTSIZES:
            generic_setinputsizes = context._prepare_set_input_sizes()

            if generic_setinputsizes:
                try:
                    dialect.do_set_input_sizes(
                        context.cursor, generic_setinputsizes, context
                    )
                except BaseException as e:
                    self._handle_dbapi_exception(
                        e, str(statement), parameters, None, context
                    )

        cursor, str_statement, parameters = (
            context.cursor,
            context.statement,
            context.parameters,
        )

        effective_parameters: Optional[_AnyExecuteParams]

        if not context.executemany:
            effective_parameters = parameters[0]
        else:
            effective_parameters = parameters

        if self._has_events or self.engine._has_events:
            for fn in self.dispatch.before_cursor_execute:
                str_statement, effective_parameters = fn(
                    self,
                    cursor,
                    str_statement,
                    effective_parameters,
                    context,
                    context.executemany,
                )

        if self._echo:
            self._log_info(str_statement)

            stats = context._get_cache_stats()

            if not self.engine.hide_parameters:
                self._log_info(
                    "[%s] %r",
                    stats,
                    sql_util._repr_params(
                        effective_parameters,
                        batches=10,
                        ismulti=context.executemany,
                    ),
                )
            else:
                self._log_info(
                    "[%s] [SQL parameters hidden due to hide_parameters=True]",
                    stats,
                )

        evt_handled: bool = False
        try:
            if context.execute_style is ExecuteStyle.EXECUTEMANY:
                effective_parameters = cast(
                    "_CoreMultiExecuteParams", effective_parameters
                )
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_executemany:
                        if fn(
                            cursor,
                            str_statement,
                            effective_parameters,
                            context,
                        ):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_executemany(
                        cursor,
                        str_statement,
                        effective_parameters,
                        context,
                    )
            elif not effective_parameters and context.no_parameters:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute_no_params:
                        if fn(cursor, str_statement, context):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_execute_no_params(
                        cursor, str_statement, context
                    )
            else:
                effective_parameters = cast(
                    "_CoreSingleExecuteParams", effective_parameters
                )
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute:
                        if fn(
                            cursor,
                            str_statement,
                            effective_parameters,
                            context,
                        ):
                            evt_handled = True
                            break
                if not evt_handled:
>                   self.dialect.do_execute(
                        cursor, str_statement, effective_parameters, context
                    )

../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1967:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/default.py:941: in do_execute
    cursor.execute(statement, parameters)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <psycopg.Cursor [closed] [IDLE] (host=localhost port=48807 user=guest database=7f251298-6147-47c9-964a-2895e7f26a05) at 0x77d2bd3c8a90>
query = 'SELECT db_dbnode_1.id, db_dbnode_1.uuid, db_dbnode_1.node_type, db_dbnode_1.process_type, db_dbnode_1.label, db_dbnod...uuid_20_4996)s::UUID, %(uuid_20_4997)s::UUID, %(uuid_20_4998)s::UUID, %(uuid_20_4999)s::UUID, %(uuid_20_5000)s::UUID))'
params = {'param_1': '%', 'uuid_10_1': '0a2d13ec-8826-41f0-960d-e77a86e2f272', 'uuid_10_10': '0de99d7d-88d4-4f2c-b9aa-005ed56ffa0c', 'uuid_10_100': '280bb0cf-f87b-4f5c-813c-c5c77d9b10fd', ...}

    def execute(
        self,
        query: Query,
        params: Params | None = None,
        *,
        prepare: bool | None = None,
        binary: bool | None = None,
    ) -> Self:
        """
        Execute a query or command to the database.
        """
        try:
            with self._conn.lock:
                self._conn.wait(
                    self._execute_gen(query, params, prepare=prepare, binary=binary)
                )
        except e._NO_TRACEBACK as ex:
>           raise ex.with_traceback(None)
E           psycopg.OperationalError: sending query and params failed: number of parameters must be between 0 and 65535

../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/psycopg/cursor.py:97: OperationalError

The above exception was the direct cause of the following exception:

self = <tests.storage.psql_dos.test_query.TestQueryBuilderParameterLimits object at 0x77d2c663da50>

    def test_large_in_filter_batching(self):
        """Test that large IN filters are automatically batched to avoid parameter limits."""
        # Use all 50k nodes - this would fail without batching
        qb = orm.QueryBuilder()
        qb.append(orm.Node, filters={'uuid': {'in': self.all_uuids}})
>       results = qb.all(flat=True)

tests/storage/psql_dos/test_query.py:78:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/aiida/orm/querybuilder.py:1126: in all
    matches = list(self.iterall(batch_size=batch_size))
src/aiida/orm/querybuilder.py:1083: in iterall
    for item in self._impl.iterall(self.as_dict(), batch_size):
src/aiida/storage/psql_dos/orm/querybuilder/main.py:210: in iterall
    for resultrow in session.execute(stmt):
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/session.py:2362: in execute
    return self._execute_internal(
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/session.py:2247: in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/context.py:293: in orm_execute_statement
    result = conn.execute(
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1418: in execute
    return meth(
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/sql/elements.py:515: in _execute_on_connection
    return connection._execute_clauseelement(
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1640: in _execute_clauseelement
    ret = self._execute_context(
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1846: in _execute_context
    return self._exec_single_context(
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1986: in _exec_single_context
    self._handle_dbapi_exception(
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py:2355: in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py:1967: in _exec_single_context
    self.dialect.do_execute(
../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/default.py:941: in do_execute
    cursor.execute(statement, parameters)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <psycopg.Cursor [closed] [IDLE] (host=localhost port=48807 user=guest database=7f251298-6147-47c9-964a-2895e7f26a05) at 0x77d2bd3c8a90>
query = 'SELECT db_dbnode_1.id, db_dbnode_1.uuid, db_dbnode_1.node_type, db_dbnode_1.process_type, db_dbnode_1.label, db_dbnod...uuid_20_4996)s::UUID, %(uuid_20_4997)s::UUID, %(uuid_20_4998)s::UUID, %(uuid_20_4999)s::UUID, %(uuid_20_5000)s::UUID))'
params = {'param_1': '%', 'uuid_10_1': '0a2d13ec-8826-41f0-960d-e77a86e2f272', 'uuid_10_10': '0de99d7d-88d4-4f2c-b9aa-005ed56ffa0c', 'uuid_10_100': '280bb0cf-f87b-4f5c-813c-c5c77d9b10fd', ...}

    def execute(
        self,
        query: Query,
        params: Params | None = None,
        *,
        prepare: bool | None = None,
        binary: bool | None = None,
    ) -> Self:
        """
        Execute a query or command to the database.
        """
        try:
            with self._conn.lock:
                self._conn.wait(
                    self._execute_gen(query, params, prepare=prepare, binary=binary)
                )
        except e._NO_TRACEBACK as ex:
>           raise ex.with_traceback(None)
E           sqlalchemy.exc.OperationalError: (psycopg.OperationalError) sending query and params failed: number of parameters must be between 0 and 65535
E           [SQL: SELECT db_dbnode_1.id, db_dbnode_1.uuid, db_dbnode_1.node_type, db_dbnode_1.process_type, db_dbnode_1.label, db_dbnode_1.description, db_dbnode_1.ctime, db_dbnode_1.mtime, db_dbnode_1.attributes, db_dbnode_1.extras, db_dbnode_1.repository_metadata, db_dbnode_1.dbcomputer_id, db_dbnode_1.user_id
E           FROM db_dbnode AS db_dbnode_1
E           WHERE CAST(db_dbnode_1.node_type AS VARCHAR) LIKE %(param_1)s::VARCHAR AND (db_dbnode_1.uuid IN (%(uuid_1_1)s::UUID, %(uuid_1_2)s::UUID, %(uuid_1_3)s::UUID, %(uuid_1_4)s::UUID, %(uuid_1_5)s::UUID, %(uuid_1_6)s::UUID, %(uuid_1_7)s::UUID, %(uuid_1_8)s::UUID, %(uuid_1_9)s::UUID, %(uuid_1_10)s::UUID, %(uuid_1_11)s::UUID, %(uuid_1_12)s::UUID, %(uuid_1_13)s::UUID, %(uuid_1_14)s::UUID, %(uuid_1_15)s::UUID, %(uuid_1_16)s::UUID, %(

and

44da-94b4-6fe494b50ed0', 'uuid_20_4986': 'd535d919-d013-4a32-9c05-a95fba2a6ea5', 'uuid_20_4987': 'd6dba2d9-88de-4b99-b854-27a9d207f155', 'uuid_20_4988': 'd885a887-8aa1-4179-bb2e-25b97c06230b', 'uuid_20_4989': 'd9b6833d-9cfa-4d61-a655-038c0ffcb6c2', 'uuid_20_4990': 'de9ee5ea-dde7-4d37-ac6c-fcf4812ce26a', 'uuid_20_4991': 'e56ba03c-8d0c-451e-911c-0d775c298537', 'uuid_20_4992': 'e87d7905-fbae-49da-bb46-17b5cba07c4d', 'uuid_20_4993': 'e9e1261b-b688-4cdf-865c-55aafb7322de', 'uuid_20_4994': 'ed941dc2-d23b-414d-a26f-5d87258b2ff3', 'uuid_20_4995': 'ef71d53f-28ee-472a-aa1e-001f95c623b2', 'uuid_20_4996': 'efbe388e-7906-4f91-9df3-22ae79476cca', 'uuid_20_4997': 'f798eb96-fb60-40a8-9486-4f8b09e3b555', 'uuid_20_4998': 'f83a8528-e7b6-483a-a194-df7cf55d81bb', 'uuid_20_4999': 'fa75665a-afec-45ad-8495-c42b565c39af', 'uuid_20_5000': 'fdec29cb-c44d-4ce3-b583-7cee07b9975e'}]
E           (Background on this error at: https://sqlalche.me/e/20/e3q8)

../../../../.aiida_venvs/aiida-dev/lib/python3.10/site-packages/psycopg/cursor.py:97: OperationalError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/psycopg/cursor.py(97)execute()
     95                 )
     96         except e._NO_TRACEBACK as ex:
---> 97             raise ex.with_traceback(None)
     98         return self
     99

ipdb> import traceback
ipdb> traceback.print_stack()
  File "/home/geiger_j/.aiida_venvs/aiida-dev/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192, in console_main
    code = main()
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/main.py", line 271, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/main.py", line 325, in _main
    config.hook.pytest_runtestloop(session=session)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
    item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/runner.py", line 133, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/runner.py", line 228, in call_and_report
    hook.pytest_exception_interact(node=item, call=call, report=report)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/debugging.py", line 299, in pytest_exception_interact
    _enter_pdb(node, call.excinfo, report)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/debugging.py", line 366, in _enter_pdb
    post_mortem(tb)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/_pytest/debugging.py", line 389, in post_mortem
    p.interaction(None, t)
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/IPython/core/debugger.py", line 443, in interaction
    OldPdb.interaction(self, frame, tb)
  File "/usr/lib/python3.10/pdb.py", line 348, in interaction
    self._cmdloop()
  File "/usr/lib/python3.10/pdb.py", line 313, in _cmdloop
    self.cmdloop()
  File "/usr/lib/python3.10/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/usr/lib/python3.10/pdb.py", line 413, in onecmd
    return cmd.Cmd.onecmd(self, line)
  File "/usr/lib/python3.10/cmd.py", line 216, in onecmd
    return self.default(line)
  File "/usr/lib/python3.10/pdb.py", line 372, in default
    exec(code, globals, locals)
  File "<stdin>", line 1, in <module>
ipdb>

Seems like batching inside the get_filter_expr_* methods does not resolve the issue. One could modify the function signature to return multiple, batched expressions, rather than one:

def get_filter_expr(self, operator, value, attr_key, **kwargs):
    """Return an iterator of filter expressions to handle large parameter lists."""
    # ... existing logic ...
    
    if operator == 'in' and len(value) > 5000:
        # Split into chunks and yield multiple expressions
        chunks = [value[i:i+5000] for i in range(0, len(value), 5000)]
        for chunk in chunks:
            if is_jsonb:
                yield self.get_filter_expr_from_jsonb('in', chunk, attr_key, **kwargs)
            else:
                yield self.get_filter_expr_from_column('in', chunk, column)
    else:
        # Single expression - wrap in generator for consistency
        if is_jsonb:
            yield self.get_filter_expr_from_jsonb(operator, value, attr_key, **kwargs)
        else:
            yield self.get_filter_expr_from_column(operator, value, column)

# In build_filters method:
for path_spec, filter_operation_dict in filter_spec.items():
    # ... existing logic ...
    for operator, value in filter_operation_dict.items():
        for expr in self.get_filter_expr(operator, value, attr_key, **kwargs):
            expressions.append(expr)

This reverts commit 8faf771.

codecov · 2025-09-02T08:20:55Z

Codecov Report

❌ Patch coverage is 23.07692% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 24.90%. Comparing base (313f342) to head (34ccac9).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...rc/aiida/storage/psql_dos/orm/querybuilder/main.py	23.08%	10 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (313f342) and HEAD (34ccac9). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (313f342) HEAD (34ccac9)

2 1

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #6998       +/-   ##
===========================================
- Coverage   79.05%   24.90%   -54.15%     
===========================================
  Files         566      566               
  Lines       43675    43683        +8     
===========================================
- Hits        34522    10873    -23649     
- Misses       9153    32810    +23657

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This reverts commit 5e091cc.

… than true temp table

This reverts commit f9a66bf.

GeigerJ2 · 2025-09-11T07:30:00Z

Closing in favor of #6993.

GeigerJ2 added 3 commits September 2, 2025 09:19

copy over test files from 6991

b6ad44e

batch in statements of get_filter_expr_from_[column|jsonb]

8faf771

Revert "batch in statements of get_filter_expr_from_[column|jsonb]"

6ca29ff

This reverts commit 8faf771.

GeigerJ2 added 10 commits September 2, 2025 10:22

Add QB test that fails without batching

9badbd1

Add two failing tests

f9a66bf

batch in statements of get_filter_expr_from_[column|jsonb]

5e091cc

Revert "batch in statements of get_filter_expr_from_[column|jsonb]"

0f5a906

This reverts commit 5e091cc.

try approach with _create_temp_table_join that uses VALUES rather…

dfd2f1a

… than true temp table

Revert "Add two failing tests"

b74713b

This reverts commit f9a66bf.

remove filter batching tests

a4e4503

re-add large archive and start writing archive export test

f6904f4

wip for _create_temp_table

e20fb2a

wip

34ccac9

GeigerJ2 mentioned this pull request Sep 3, 2025

Fix PSQL OperationalError on archive creation #6993

Open

GeigerJ2 closed this Sep 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix PSQL OperationalError on archive creation at QB level #6998

Fix PSQL OperationalError on archive creation at QB level #6998

Uh oh!

GeigerJ2 commented Sep 2, 2025 •

edited

Loading

Uh oh!

codecov bot commented Sep 2, 2025 •

edited

Loading

Uh oh!

GeigerJ2 commented Sep 11, 2025

Uh oh!

Uh oh!

Fix PSQL OperationalError on archive creation at QB level #6998

Fix PSQL OperationalError on archive creation at QB level #6998

Uh oh!

Conversation

GeigerJ2 commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

GeigerJ2 commented Sep 11, 2025

Uh oh!

Uh oh!

GeigerJ2 commented Sep 2, 2025 •

edited

Loading

codecov bot commented Sep 2, 2025 •

edited

Loading