Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration tests issue with logging #7926

Open
aldbr opened this issue Dec 4, 2024 · 0 comments
Open

Integration tests issue with logging #7926

aldbr opened this issue Dec 4, 2024 · 0 comments

Comments

@aldbr
Copy link
Contributor

aldbr commented Dec 4, 2024

Reported in #7924

As I said, I am not sure to fully understand, but it looks like it is due to the weakref. When we are in __deleteTQIfEmpty, it's like the streams have already been cleaned up, and the logger, even if it's still alive, cannot emit the messages anymore.

Quick and easy but a bit ugly fixes would consist in:

  • removing the log statements from __deleteTQIfEmpty, not a good idea
  • wrap the log statements in try/except blocks

Should we take the time to investigate the issue? I am not so sure as it doesn't seem to happen in production (probably just in the tests).

--- Logging error ---
Traceback (most recent call last):
  File "/home/dirac/ServerInstallDIR/diracos/lib/python3.11/logging/__init__.py", line 1113, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/home/dirac/ServerInstallDIR/diracos/lib/python3.11/weakref.py", line 666, in _exitfunc
    f()
  File "/home/dirac/ServerInstallDIR/diracos/lib/python3.11/weakref.py", line 590, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/home/dirac/ServerInstallDIR/diracos/lib/python3.11/site-packages/DIRAC/Core/Utilities/DictCache.py", line 246, in _purgeAll
    deleteFunction(cache[cKey]["value"])
  File "/home/dirac/ServerInstallDIR/diracos/lib/python3.11/site-packages/DIRAC/WorkloadManagementSystem/DB/TaskQueueDB.py", line 950, in __deleteTQIfEmpty
    result = self.deleteTaskQueueIfEmpty(tqId, tqOwner, tqOwnerGroup)
  File "/home/dirac/ServerInstallDIR/diracos/lib/python3.11/site-packages/DIRAC/WorkloadManagementSystem/DB/TaskQueueDB.py", line 976, in deleteTaskQueueIfEmpty
    retVal = self._query(sqlCmd, conn=connObj)
  File "/home/dirac/ServerInstallDIR/diracos/lib/python3.11/site-packages/DIRAC/Core/Utilities/MySQL.py", line 718, in _query
    self.log.debug(f"_query: {self._safeCmd(cmd)}")
  File "/home/dirac/ServerInstallDIR/diracos/lib/python3.11/site-packages/DIRAC/FrameworkSystem/private/standardLogging/Logging.py", line [340](https://github.com/DIRACGrid/DIRAC/actions/runs/11820090778/job/32931577624#step:12:341), in debug
    return self._createLogRecord(LogLevels.DEBUG, sMsg, sVarMsg)
  File "/home/dirac/ServerInstallDIR/diracos/lib/python3.11/site-packages/DIRAC/FrameworkSystem/private/standardLogging/Logging.py", line 407, in _createLogRecord
    self._logger.log(level, "%s", sMsg, exc_info=exc_info, extra=extra)
Message: '%s'
Arguments: ('_query: SELECT TQId FROM `tq_TaskQueues` WHERE Enabled >= 1 AND `tq_TaskQueues`.TQId = 2 AND `tq_TaskQueues`.TQId not in ( SELECT DISTINCT TQId from `tq_Jobs` )',)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant