How to configure the JVM, resource groups, and query to prevent worker restarts due to OutOfMemoryError (OOM)?

[A-little-bit-of-data]

trino version : 430
The operating environment is Kubernetes.

resource group configuration

{
	"rootGroups": [
	  {
		"name": "global",
		"softMemoryLimit": "80%",
		"hardConcurrencyLimit": 30,
		"maxQueued": 1000,
		"schedulingPolicy": "weighted",
		"jmxExport": true,
		"subGroups": [
		  {
			"name": "amluser",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 5,
			"maxQueued": 100,
			"schedulingWeight": 1
		  },
		  {
			"name": "a_read",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 5,
			"maxQueued": 100,
			"schedulingWeight": 1
		  },
		  {
			"name": "b_read",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 5,
			"maxQueued": 100,
			"schedulingWeight": 1
		  },
		  {
			"name": "c_read",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 5,
			"maxQueued": 150,
			"schedulingWeight": 1
		  },
		  {
			"name": "d_read",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 5,
			"maxQueued": 100,
			"schedulingWeight": 1
		  },
		  {
			"name": "e",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 10,
			"maxQueued": 600,
			"schedulingWeight": 1
		  },
		  {
			"name": "f_read",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 5,
			"maxQueued": 600,
			"schedulingWeight": 1
		  },
		  {
			"name": "g_read_bi",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 5,
			"maxQueued": 100,
			"schedulingWeight": 1
		  },
		  {
			"name": "readonly",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 5,
			"maxQueued": 100,
			"schedulingWeight": 1
		  },
		  {
			"name": "huser",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 5,
			"maxQueued": 100,
			"schedulingWeight": 1
		  },
		  {
			"name": "iuser",
			"softMemoryLimit": "80%",
			"hardConcurrencyLimit": 5,
			"maxQueued": 100,
			"schedulingWeight": 1
		  }
		]
	  },
	  {
		"name": "admin",
		"softMemoryLimit": "100%",
		"hardConcurrencyLimit": 50,
		"maxQueued": 100,
		"schedulingPolicy": "query_priority",
		"jmxExport": true
	  }
	],
	"selectors": [
	  {
		"user": "amluser",
		"group": "global.amluser"
	  },
	  {
		"user": "a_read",
		"group": "global.a_read"
	  },
	  {
		"user": "b_read",
		"group": "global.b_read"
	  },
	  {
		"user": "c_read",
		"group": "global.c_read"
	  },
	  {
		"user": "d_read",
		"group": "global.d_read"
	  },
	  {
		"user": "e",
		"group": "global.e"
	  },
	  {
		"user": "f_read",
		"group": "global.f_read"
	  },
	  {
		"user": "g_read_bi",
		"group": "global.g_read_bi"
	  },
	  {
		"user": "readonly",
		"group": "global.readonly"
	  },
	  {
		"user": "huser",
		"group": "global.huser"
	  },
	  {
		"user": "iuser",
		"group": "global.iuser"
	  },
	  {
		"user": "trino_admin",
		"group": "admin"
	  }
	],
	"cpuQuotaPeriod": "1h"
  }

trino coordinator and worker deploy config

trino coordinator:

config.properties: |-
    coordinator=true
    node-scheduler.include-coordinator=false
    http-server.http.port=8080
    query.max-memory=98GB
    query.max-memory-per-node=93GB
    memory.heap-headroom-per-node=5GB

jvm.config: |-
    -server
    -agentpath:/usr/lib/trino/bin/libjvmkill.so
    -Xmx98G
    -XX:+UseG1GC
    -XX:G1HeapRegionSize=32M
    -XX:+ExplicitGCInvokesConcurrent
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:+ExitOnOutOfMemoryError
    -XX:-OmitStackTraceInFastThrow
    -XX:ReservedCodeCacheSize=512M
    -XX:PerMethodRecompilationCutoff=10000
    -XX:PerBytecodeRecompilationCutoff=10000
    -Djdk.attach.allowAttachSelf=true
    -Djdk.nio.maxCachedBufferSize=2000000
    -XX:+UnlockDiagnosticVMOptions
    # Reduce starvation of threads by GClocker, recommend to set about the number of cpu cores (JDK-8192647)
    -XX:GCLockerRetryAllocationCount=32
    -XX:G1PeriodicGCInterval=10000
    -Duser.timezone=Etc/GMT-8

trino worker:

config.properties: |-
    coordinator=false
    http-server.http.port=8080
    query.max-memory=98GB
    query.max-memory-per-node=93GB
    memory.heap-headroom-per-node=5GB

  jvm.config: |-
    -server
    -agentpath:/usr/lib/trino/bin/libjvmkill.so
    -Xmx98G
    -XX:+UseG1GC
    -XX:G1HeapRegionSize=32M
    -XX:+ExplicitGCInvokesConcurrent
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:+ExitOnOutOfMemoryError
    -XX:-OmitStackTraceInFastThrow
    -XX:ReservedCodeCacheSize=512M
    -XX:PerMethodRecompilationCutoff=10000
    -XX:PerBytecodeRecompilationCutoff=10000
    -Djdk.attach.allowAttachSelf=true
    -Djdk.nio.maxCachedBufferSize=2000000
    -XX:+UnlockDiagnosticVMOptions
    # Reduce starvation of threads by GClocker, recommend to set about the number of cpu cores (JDK-8192647)
    -XX:GCLockerRetryAllocationCount=32
    -XX:G1PeriodicGCInterval=10000
    -Duser.timezone=Etc/GMT-8

My current configuration is as shown above. I recently deployed some services, but I frequently encounter worker out-of-memory (OOM) errors, as shown in the following images.

Looking at the worker memory monitoring graph, the peak times are basically at 2 AM, 6 AM, 8 AM, and 8 PM, which are the peak business periods. We dare not reduce the concurrency any further at the moment. Are there any problems with the above configuration, or are there any areas that need to be optimized?

[A-little-bit-of-data]

@hashhar Sorry to bother you, but could you please help me out with my problem when you have some free time? I also consulted GPT and they suggested improvements to my configuration, such as adding -XX:MaxDirectMemorySize and -XX:MaxMetaspaceSize to the JVM. Thank you.