java.lang.OutOfMemoryError: Java heap space

[anjalijain22]

Hello, hope you're doing well!

We have been running into a java.lang.OutOfMemoryError: Java heap space error message while loading data into seqr (Hail backend). Generally, resubmitting the pipeline solves the issue but ideally we would like to prevent this from happening if there is a setting we can change on our end. The virtual machine seems to have plenty of memory available so we're not sure why we keep running into this issue.

[91462.786s][warning][gc,alloc] dispatcher-event-loop-11: Retried waiting for GCLocker too often allocating 280066 words
Exception in thread "dispatcher-event-loop-11" java.lang.OutOfMemoryError: Java heap space
 at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
 at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
 at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
 at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
 at org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
 at java.base/java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1873)
 at java.base/java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1844)
...

Many thanks in advance for your help!

Best,
Anjali

[hanars]

seqr has a couple of environment variables you can set in your helm configuration that will tell the hail search backend explicitly how much memory to use - for example

hail-search:
    environment:
      MACHINE_MEM: "96"
      JAVA_OPTS_XSS: "16M"

The maximum MACHINE_MEM should be the actual available memory on the machine where hail search is running
The JAVA_OPTS_XSS variable sets the thread stack size for the java threads. You want to ensure that this number multiplied by the number of available CPUs does not exceed your available machine memory. We also recommend that JAVA_OPTS_XSS * <num_cpus> has some buffer left and does not come to the total available memory.
We are unable to make more detailed recommendations for the best values for these variables in your deployment, but recommend you play around with these setting to see what helps

[bpblanken]

You may also try helm settings specific to the pipeline-runner:

pipeline-runner:
  enabled: true
  environment:
    PYSPARK_SUBMIT_ARGS='--driver-memory 16G --executor-memory 16g pyspark-shell'

to increase the ram available to hail during pipeline runs.