Exceeding memory for Express_Run387696_StreamHLTMonitor merge

[mmusich]

This issue is a mirror of cmsTalk post.

We have an express merge job that exceeds the 2 GB memory limit for the Express_Run387696_StreamHLTMonitor workflow.
Run 387696 is very long (17h:48min, ~ 2750 LS), and such issues could arise for long runs but should be investigated.

The tarball can be found here:

/eos/user/c/cmst0/public/PausedJobs/Run2024J/maxMemory/Express_Run387696_StreamHLTMonitor

Marco (as ORM)

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @mmusich.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign dqm

[cmsbuild]

New categories assigned: dqm

@antoniovagnerini,@rseidita you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

Based on the log the job seems to be effectively DQM harvesting. The job opened 1192 files before being signaled to shut down.

We've seen before the harvesting memory requirements scale with the number of input files (e.g. #38976, https://cms-talk.web.cern.ch/t/re-2018-replay-for-pps-pcl-test-dqm-expresmergewrite-memory-too-high/6114/8)