Documentation and configuration updates from JGroups retest (#520)

keycloak · Sep 19, 2023 · 3bc2219 · 3bc2219
1 parent 15bdeca
commit 3bc2219
Show file tree

Hide file tree

Showing 4 changed files with 6 additions and 2 deletions.
diff --git a/doc/kubernetes/modules/ROOT/pages/error-messages.adoc b/doc/kubernetes/modules/ROOT/pages/error-messages.adoc
@@ -126,6 +126,7 @@ org.jgroups.util.ThreadPool`: thread pool is full (max=xx, active=xx); thread du
 
 Context::
 This happens when the thread pool in JGroups runs out of threads.
+For this message to appear with the default JGroups configuration, the system property `jgroups.thread_dumps_threshold` needs to be set to `1`, as otherwise the message will appear only after 10000 threads have been rejected.
 
 Cause::
 Each Keycloak pod has executor threads that run the RESTEasy requests.

diff --git a/doc/kubernetes/modules/ROOT/pages/load-behavior.adoc b/doc/kubernetes/modules/ROOT/pages/load-behavior.adoc
@@ -33,6 +33,7 @@ With the number of threads in the executor pool being an order of magnitude larg
 +
 The combined number of executor threads in all Keycloak nodes in the cluster shouldn't exceed the number of threads available in JGroups thread pool to avoid the error described in
 xref:kubernetes-guide::error-messages.adoc#jgroups-thread-pool-is-full[`org.jgroups.util.ThreadPool: thread pool is full`].
+To see the error the first time it happens, the system property `jgroups.thread_dumps_threshold` needs to be set to `1`, as otherwise the message will appear only after 10000 threads have been rejected.
 +
 --
 include::partial$executor-jgroups-thread-calculation.adoc[]

diff --git a/doc/kubernetes/modules/ROOT/partials/executor-jgroups-thread-calculation.adoc b/doc/kubernetes/modules/ROOT/partials/executor-jgroups-thread-calculation.adoc
@@ -1,2 +1,3 @@
-Due to the bug https://issues.redhat.com/browse/ISPN-14780[ISPN-14780] there was currently no known way to configure the size different from the default JGroups 100 max threads before Keycloak 23.0.0 and 22.0.2, so assuming a Keycloak cluster with 4 Pods, each Pod shouldn't have more than 25 worker threads.
+With https://issues.redhat.com/browse/ISPN-14780[ISPN-14780] being fixed in Keycloak 22.0.2, the number of JGroup threads is `200` by default, and can be configured using the property Java system property `jgroups.thread_pool.max_threads`.
+As shown in experiments, assuming a Keycloak cluster with 4 Pods, each Pod shouldn't have more than 50 worker threads so that it doesn't run out of threads in the JGroup thread pool of 200.
 Use the Quarkus configuration options `quarkus.thread-pool.max-threads` to configure the maximum number of worker threads.
diff --git a/provision/minikube/keycloak/templates/keycloak.yaml b/provision/minikube/keycloak/templates/keycloak.yaml
@@ -204,10 +204,11 @@ spec:
                 value: "10000"
 {{ end }}
               - name: JAVA_OPTS_APPEND
-                # using non-blocking random, make DNS lookups expire after 10 seconds and not cache them forever
+                # jgroups.thread_dumps_threshold ensure that a log message "thread pool is full" appears once the thread pool is full for the first time
                 value: >
                   -Xms{{ .Values.heapInitMB }}m -Xmx{{ .Values.heapMaxMB }}m
                   -XX:MetaspaceSize={{ .Values.metaspaceInitMB }}m -XX:MaxMetaspaceSize={{ .Values.metaspaceMaxMB }}m
+                  -Djgroups.thread_dumps_threshold=1
                   -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8787
 {{- if .Values.otel }}
                   -javaagent:/otel/opentelemetry-javaagent-{{ .Values.otelVersion }}.jar