Fix problem that Nextflow Executions stuck on Kubernetes

[Lehmann-Fabian]

When running Nextflow workflows on Kubernetes, workflows would often get stuck with Pods remaining in the Succeeded state without being removed, and the workflow itself would not progress.

After running a jstack on the Nextflow Java process, I identified a thread hanging at this line in TraceRecord.groovy:
TraceRecord.groovy#L422.

To resolve this, I replaced the Groovy methods with native Java functions. After using this change in production for over six months, the problem has not recurred, so I am contributing it with confidence that it will benefit other Nextflow and Kubernetes users.

We experienced this issue on three different clusters, all running Ceph with Kubernetes.

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	c2faac1
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/6734cd1d2d9e6c00085ae062

[bentsherman]

That line should be delegating to this method

It is slightly different from your approach, maybe it is getting into an infinite loop

[Lehmann-Fabian]

Yes, and the BufferedReader is created by: Files.newBufferedReader(self, Charset.defaultCharset());.
The jstack trace is always in this line: IOGroovyMethods.java#L862
There may be a timeout that is not caught in the distributed file system as an infinite loop would cause the answer StringBuilder to fill until OOM.
Anyway, the new implementation does not have the problem and is even a little bit more memory efficient, as it reads the trace file line by line ;)