Nightly builds are broken

[pjfanning]

We run tests nightly and they are now timing out every night. Something appears to be hanging.
https://github.com/apache/pekko/actions/workflows/nightly-builds.yml

The tests failed on July 14 after passing on July 13 but have hung every night since July 15.

We are seeing tests taking crazy long times to complete

[07-22 05:19:55.084] [info] - must work with a massive chain of maps (4 hours, 7 minutes, 49 seconds)

The big issues appear to start when org.apache.pekko.stream.impl.fusing.InterpreterStressSpec runs. After that starts, most of the logging is thread dumps and GC stats. Very few tests appear to complete.

[pjfanning]

must work with a massive chain of maps used to pass in more like 37 seconds, 148 milliseconds and now takes over 4hrs

There is a good chance that InterpreterStressSpec or some test that starts earlier in the test cycle has caused threads or other important resources to crash causing the Pekko processing to slow down drastically or hang.

[He-Pin]

https://github.com/apache/pekko/actions/runs/29296351052/job/86970609999

[pjfanning]

Changes that were merged after last successful run. I'm excluding some changes that are more doc or build oriented.

• Fix ManualTime.expectNoMessageFor async delivery race #3329
• Replace java.lang.reflect with MethodHandle/VarHandle across core modules #3300
• Fix reentrant BatchingExecutor execution #3331
• fix: enable strict-unsealed-patmat lint and fix exhaustiveness warnings #3330
• Fix deferred TLS output handling on inbound close #3324
• fix: release references to completed stage logics in GraphInterpreter #3030
• Fix deferred TLS output handling on inbound close #3324
• Check if expected messages are produced in FlowLimitSpec #3351
• use lambdas instead of new Runnable #3349
  Issues were probably already happening after those changes (July 13 and 14) but it's possible that changes on July 15 (not listed) may be the issue.

[He-Pin]

Let me check this, I think it around GraphInterpreter.scala

[pjfanning]

https://github.com/apache/pekko/actions/runs/29296351052/job/86970609999

@He-Pin that run may have been just cancelled by GitHub. We have more recent runs that are explicitly cancelled after 6 hours which seems to be a limit for how long we are allowed to run jobs. These jobs should complete after about an hour.

[He-Pin]

#3030 I think this is the cause

[pjfanning]

I dropped a tag on that 3030 commit (by coincidence) and am running the nightly tests from that tag to see if the issue exists at that tag point.

[He-Pin]

I'm preparing a fix