LogOutputs inspection

[fstagni]

simply run

# Old command, not working so well
# grep 'ERROR\|EXCEPT' runit/*/*/log/current | cut -f 1,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20 -d ' ' | sort | uniq -c | grep $(date +%F) | sort -nr | head -n 20
grep $(date +%F) runit/*/*/log/current | grep 'ERROR\|EXCEPT' | sed -E 's@[a-z,A-Z,/]+:.*Z (.*)@\1@g' | sort | uniq -c | sort -nr | head -n 20

to get the most frequent errors, then start investigating them.

Also, centralized logging (only services for now)

https://es-dirac-certification.cern.ch

• The link above now has to be os-dirac, not es-dirac
• Bunch of errors for submitting to various sistes
• VOMS2CS agent has errors connecting to VOMS
• Web issues (below): Fixes v9 WebAppDIRAC#772

[fstagni]

Also:

oxyManager (61.91 ms) ERROR 500: Internal Server Error
2025-02-13T10:36:05,137985Z WebApp/Authorization [139687702234688] DEBUG: Trying to authenticate DN=/C=PL/O=GRID/O=NCBJ/CN=Henryk Giemza group=dirac_user
2025-02-13T10:36:05,138172Z WebApp/Authorization [139687702234688] DEBUG: Using hardcoded properties for method generatePlot : ['all']
2025-02-13T10:36:05,139056Z WebApp/AccountingHandler [139688537249344] NOTICE: Incoming request (192.68.51.225:58428)[dirac_user:hgiemza] Accounting: generateP
    "ExpirationTime": str(record[3]),
                          ~~~~~~^^^

  IndexError: list index out of range

2025-02-13T10:35:57,564195Z WebApp/ProxyManagerHandler [139687733753408] ERROR: Exception serving request list index out of range:IndexError('list index out of range')
Traceback (most recent call last):
  File "/opt/dirac/versions/v9.0.0a42-1739368301/Linux-x86_64/lib/python3.11/site-packages/DIRAC/Core/Tornado/Server/private/BaseRequestHandler.py", line 642,
in _executeMethod
    return self.methodObj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/dirac/versions/v9.0.0a42-1739368301/Linux-x86_64/lib/python3.11/site-packages/WebAppDIRAC/WebApp/handler/ProxyManagerHandler.py", line 77, in web_
getProxyManagerData
    "ExpirationTime": str(record[3]),
                          ~~~~~~^^^

[andresailer]

      1 runit/Configuration/VOMS2Configuration/VOMS2CSAgent/VOMS2CSSynchronizer ERROR: Could not retrieve user information Unauthorized query ( 1111 : Failed to contact the VOMS server: https://voms-dteam-auth.cern.ch:8443/voms/dteam/apiv2/users:ConnectionError(MaxRetryError("HTTPSConnectionPool(host='voms-dteam-auth.cern.ch', port=8443): Max retries exceeded with url: /voms/dteam/apiv2/users?startIndex=0&pageSize=

e.g.

2025-02-13T10:29:07,778585Z WorkloadManagement/SiteDirectorDteam/WorkloadManagement/SiteDirectorDteam ERROR: Failed submission to queue Queue ce504.cern.ch_condor:
Command ['condor_submit', '-terse', '-pool', 'ce504.cern.ch:9619', '-remote', 'ce504.cern.ch', '/opt/dirac/data/HTCondor/work/HTCondorCE_ypahb_pa.sub'] failed with: 1 - ERROR: Failed to connect to queue manager ce504.cern.ch
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using SCITOKENS
2025-02-13T10:28:49,934084Z WorkloadManagement/SiteDirectorDteam/WorkloadManagement/SiteDirectorDteam ERROR: The following errors occurred during the pilot submission operation Command ['condor_submit', '-terse', '-pool', 'ce514.cern.ch:9619', '-remote', 'ce514.cern.ch', '/opt/dirac/data/HTCondor/work/HTCondorCE_voytkt3u.sub'] failed with: 1 - ERROR: Failed to connect to queue manager ce514.cern.ch
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using SCITOKENS
Command ['condor_submit', '-terse', '-pool', 'ce503.cern.ch:9619', '-remote', 'ce503.cern.ch', '/opt/dirac/data/HTCondor/work/HTCondorCE_csm4p4sv.sub'] failed with: 1 - ERROR: Failed to connect to queue manager ce503.cern.ch
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using SCITOKENS
Command ['condor_submit', '-terse', '-pool', 'htcondor-ce-1-kit.gridka.de:9619', '-remote', 'htcondor-ce-1-kit.gridka.de', '/opt/dirac/data/HTCondor/work/HTCondorCE_djkedgr1.sub'] failed with: 1 - ERROR: Failed to connect to queue manager htcondor-ce-1-kit.gridka.de
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using SSL
AUTHENTICATE:1004:Failed to authenticate using SCITOKENS
AUTHENTICATE:1004:Failed to authenticate using IDTOKENS
AUTHENTICATE:1004:Failed to authenticate using FS
Command ['condor_submit', '-terse', '-pool', 'htcondor-ce-3-kit.gridka.de:9619', '-remote', 'htcondor-ce-3-kit.gridka.de', '/opt/dirac/data/HTCondor/work/HTCondorCE_mpp8os_o.sub'] failed with: 1 - ERROR: Failed to connect to queue manager htcondor-ce-3-kit.gridka.de
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using SSL
AUTHENTICATE:1004:Failed to authenticate using SCITOKENS
AUTHENTICATE:1004:Failed to authenticate using IDTOKENS
AUTHENTICATE:1004:Failed to authenticate using FS
Command ['condor_submit', '-terse', '-pool', 'htcondor-ce-2-kit.gridka.de:9619', '-remote', 'htcondor-ce-2-kit.gridka.de', '/opt/dirac/data/HTCondor/work/HTCondorCE_jgptemyh.sub'] failed with: 1 - ERROR: Failed to connect to queue manager htcondor-ce-2-kit.gridka.de
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using SSL
AUTHENTICATE:1004:Failed to authenticate using SCITOKENS
AUTHENTICATE:1004:Failed to authenticate using IDTOKENS
AUTHENTICATE:1004:Failed to authenticate using FS
Command ['condor_submit', '-terse', '-pool', 'htcondor-ce-4-kit.gridka.de:9619', '-remote', 'htcondor-ce-4-kit.gridka.de', '/opt/dirac/data/HTCondor/work/HTCondorCE_3pd6nb39.sub'] failed with: 1 - ERROR: Failed to connect to queue manager htcondor-ce-4-kit.gridka.de
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using SSL
AUTHENTICATE:1004:Failed to authenticate using SCITOKENS
AUTHENTICATE:1004:Failed to authenticate using IDTOKENS
AUTHENTICATE:1004:Failed to authenticate using FS