-
Notifications
You must be signed in to change notification settings - Fork 3k
Stop using named (with counter) VT names due to https://bugs.openjdk.org/browse/JDK-8372410 #51223
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
franz1981
commented
Nov 25, 2025
- Fixes RunOnVirtualThread has some unexpected cost due to the default VT builder #51201
ozangunalp
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we remove the counter we should remove the trailing dash in VirtualThreadsConfig#namePrefix default value. So the default thread name would be quarkus-virtual-thread.
Or we can remove the name call completely and let VTs have default thread names, until the JDK bug is resolved.
This comment has been minimized.
This comment has been minimized.
|
@ozangunalp |
|
I agree let's remove the trailing dash from |
fb2d5ea to
daf2b3e
Compare
|
I saw few failing tests before, let's see if I forgot anything. |
|
What does it bring exactly? It makes so much of a difference that we can’t wait for the JDK fix? |
That's a good point. Setting the config Edit: Like this : https://quarkus.io/guides/virtual-threads#virtual-thread-names |
gastaldi
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't forget to also change the guide here:
| Quarkus managed virtual threads are named and prefixed with `quarkus-virtual-thread-`. |
Let me show you... package red.hat.puzzles.loom;
import org.openjdk.jmh.annotations.*;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Measurement(iterations = 10, time = 400, timeUnit = TimeUnit.MILLISECONDS)
@Warmup(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
@State(Scope.Thread)
@Fork(2)
public class VirtualThreadThreadCreate {
ThreadFactory namingFactory;
ThreadFactory delegatingFactory;
ThreadFactory prefixOnlyFactory;
private String prefix;
@Setup
public void init() {
prefix = "quarkus-virtual-thread-";
namingFactory = Thread.ofVirtual().name(prefix, 0).factory();
var vtFactory = Thread.ofVirtual().factory();
final AtomicLong counter = new AtomicLong();
delegatingFactory = r -> {
Thread t = vtFactory.newThread(r);
t.setName(prefix + counter.getAndIncrement());
return t;
};
prefixOnlyFactory = Thread.ofVirtual().name(prefix).factory();
}
@Benchmark
public Thread createWithNamingFactory() {
return namingFactory.newThread(() -> {
});
}
@Benchmark
public Thread createWithDelegatingFactory() {
return delegatingFactory.newThread(() -> {
});
}
@Benchmark
public Thread createWithPrefixOnlyFactory() {
return prefixOnlyFactory.newThread(() -> {
});
}
}we got: which shows that the version without counter is not only faster but cheaper in term of allocations. |
This comment has been minimized.
This comment has been minimized.
|
🎊 PR Preview 5e1c2f0 has been successfully built and deployed to https://quarkus-pr-main-51223-preview.surge.sh/version/main/guides/
|
This comment has been minimized.
This comment has been minimized.
|
@franz1981 I'm afraid there are several tests that will need to be adapted |
|
Thanks, looking into it |
|
While waiting the status of the tests, I see these options here:
I have to be honest that although I'm usually all in for low risk changes, Loom adoption is still at early stages, and I prefer to not maintaining features which imply a cost on our side and no improvement for the life of users. |
Status for workflow
|
|
I like option 1 |
|
+1. Let's go with option 1. |
|
I actually liked the fact that virtual threads have a name with a unique number too. And I would showcase that with some demos in my presentations too 😄 This is quite useful when you are inspecting what is happening and to check where you code is executed. Just build this branch locally and with this change everything that runs on a virtual threads shows as |
This looks more a problem of logging instead, which shouldn't rely on thread names but on thread ids Thread.toString is making use of both informations because the name alone, being just a String, is not expected/enforced to identify uniquely them, whilst the thread id has been implemented to do so @ozangunalp any idea where it happens? |
If we somehow can make the logging output the thread id that would be good I think to be able to correctly identify them. |
Status for workflow
|
| Status | Name | Step | Failures | Logs | Raw logs | Build scan |
|---|---|---|---|---|---|---|
| ✔️ | JVM Tests - JDK 17 | Logs | Raw logs | 🚧 | ||
| ❌ | JVM Tests - JDK 21 | Build |
Failures | Logs | Raw logs | 🚧 |
| ✔️ | JVM Tests - JDK 21 Semeru | Logs | Raw logs | 🚧 | ||
| ✔️ | JVM Tests - JDK 25 | Logs | Raw logs | 🚧 | ||
| ✔️ | JVM Integration Tests - JDK 17 | Logs | Raw logs | 🚧 | ||
| ✔️ | JVM Integration Tests - JDK 17 Windows | Logs | Raw logs | 🚧 | ||
| ✔️ | JVM Integration Tests - JDK 21 | Logs | Raw logs | 🚧 | ||
| ❌ | JVM Integration Tests - JDK 21 Semeru | Build |
Failures | Logs | Raw logs | 🚧 |
| ✔️ | JVM Integration Tests - JDK 25 | Logs | Raw logs | 🚧 |
Full information is available in the Build summary check run.
You can consult the Develocity build scans.
Failures
⚙️ JVM Tests - JDK 21 #
- Failing: extensions/panache/hibernate-orm-rest-data-panache/deployment
📦 extensions/panache/hibernate-orm-rest-data-panache/deployment
❌ io.quarkus.hibernate.orm.rest.data.panache.deployment.build.BuildConditionsWithResourceDisabledTest. - History - More details - Source on GitHub
java.lang.RuntimeException: Failed to start quarkus
at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
at io.quarkus.runtime.Application.start(Application.java:101)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at io.quarkus.runner.bootstrap.StartupActionImpl.run(StartupActionImpl.java:350)
at io.quarkus.test.QuarkusUnitTest.beforeAll(QuarkusUnitTest.java:703)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: Metaspace
⚙️ JVM Integration Tests - JDK 21 Semeru #
- Failing: integration-tests/virtual-threads/grpc-virtual-threads
📦 integration-tests/virtual-threads/grpc-virtual-threads
❌ io.quarkus.grpc.example.streaming.VertxVirtualThreadTest.testUnary - History - More details - Source on GitHub
io.grpc.StatusRuntimeException: INTERNAL: Half-closed without a request
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:368)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:349)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:174)
at io.grpc.testing.integration.TestServiceGrpc$TestServiceBlockingStub.unaryCall(TestServiceGrpc.java:277)
at io.quarkus.grpc.example.streaming.VirtualThreadTestBase.testUnary(VirtualThreadTestBase.java:33)
at java.base/java.lang.reflect.Method.invoke(Method.java:586)
at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:995)
❌ io.quarkus.grpc.example.streaming.VirtualThreadTest. - History - More details - Source on GitHub
org.junit.jupiter.engine.execution.ConditionEvaluationException: Failed to evaluate condition [io.quarkus.test.junit.QuarkusTestExtension]: Internal error: Test class was loaded with an unexpected classloader (QuarkusClassLoader:Quarkus Base Runtime ClassLoader: TEST for JUnitQuarkusTest-no-profile (QuarkusTest)@81f4b324) or the thread context classloader (jdk.internal.loader.ClassLoaders$AppClassLoader@430764a7) was incorrect.
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1685)
at java.base/java.util.stream.Referen...
❌ io.quarkus.grpc.example.streaming.VirtualThreadTest.testGrpcClient - History - More details - Source on GitHub
java.lang.AssertionError:
1 expectation failed.
Expected status code <200> but was <500>.
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:73)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:108)
Flaky tests - Develocity
⚙️ JVM Tests - JDK 25
📦 extensions/smallrye-graphql/deployment
❌ io.quarkus.smallrye.graphql.deployment.CompletionStageTest.testSourcePost - History
1 expectation failed. Expected status code <200> but was <500>.-java.lang.AssertionError
java.lang.AssertionError:
1 expectation failed.
Expected status code <200> but was <500>.
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:73)
📦 extensions/smallrye-reactive-messaging/deployment
❌ io.quarkus.smallrye.reactivemessaging.hotreload.ConnectorChangeTest.testUpdatingConnector - History
Expecting actual: ["-6","-7","-9","-10","-11","-12","-13","-14"] to start with: ["-6", "-7", "-8", "-9"]-java.lang.AssertionError
java.lang.AssertionError:
Expecting actual:
["-6","-7","-9","-10","-11","-12","-13","-14"]
to start with:
["-6", "-7", "-8", "-9"]
at io.quarkus.smallrye.reactivemessaging.hotreload.ConnectorChangeTest.testUpdatingConnector(ConnectorChangeTest.java:41)
⚙️ JVM Tests - JDK 17 Windows
📦 extensions/micrometer-opentelemetry/deployment
❌ io.quarkus.micrometer.opentelemetry.deployment.compatibility.MicrometerTimedInterceptorTest.testTimeMethod_AsyncFailed - History
Stream has no elements-java.lang.IllegalArgumentException
java.lang.IllegalArgumentException: Stream has no elements
at io.quarkus.micrometer.opentelemetry.deployment.common.MetricDataFilter.lambda$lastReading$2(MetricDataFilter.java:213)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at io.quarkus.micrometer.opentelemetry.deployment.common.MetricDataFilter.lastReading(MetricDataFilter.java:213)
at io.quarkus.micrometer.opentelemetry.deployment.common.MetricDataFilter.lastReadingDataPoint(MetricDataFilter.java:231)
at io.quarkus.micrometer.opentelemetry.deployment.compatibility.MicrometerTimedInterceptorTest.testTimeMethod_AsyncFailed(MicrometerTimedInterceptorTest.java:150)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at io.quarkus.test.QuarkusUnitTest.runExtensionMethod(QuarkusUnitTest.java:532)
⚙️ MicroProfile TCKs Tests
📦 tcks/microprofile-lra
❌ org.eclipse.microprofile.lra.tck.TckRecoveryTests.testCancelWhenParticipantIsUnavailable - History
Expecting the metric Compensated callback was called Expected: a value equal to or greater than <1> but: <0> was less than <1>-java.lang.AssertionError
java.lang.AssertionError:
Expecting the metric Compensated callback was called
Expected: a value equal to or greater than <1>
but: <0> was less than <1>
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.eclipse.microprofile.lra.tck.TckRecoveryTests.assertMetricCallbackCalled(TckRecoveryTests.java:210)
at org.eclipse.microprofile.lra.tck.TckRecoveryTests.testCancelWhenParticipantIsUnavailable(TckRecoveryTests.java:195)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
|
The issue is most users already use the thread name in their logs to identify threads. I think we accept that downside, and can add to the documentation that VTs can still be identified via thread ids. |
Thanks @wjglerum ! I think this is really important feedback - I naively assumed from @franz1981 's comments above about using I agree this would be a substantial set back if existing log formats wouldn't be able to distinguish two different threads - in this case my preference changes, I don't think we can break that - error diagnostics needs to remain useable. I see two options:
|
|
Due to the better and correct usage of thread Ids I would vote for
No idea where loggings happen - any pointer? |
|
@franz1981 After all these discussions, I think we should go with the custom delegating thread factory approach. |
We propose default formats for logging, which are configured via |
so my idea is to just perform the composition myself there...e.g. concatenating the thread name with the thread id there. |
|
I agree that it is sensible to include the thread id in the logs. But the user can already do that by configuring the log format. I think what we should do in this PR is :
So we keep the current generally recognized behaviour, minimise the impact of the JDK bug, and give the option to name threads without counter and rely on the thread id. Other thread factories in Quarkus do have counter-based thread names: Vertx threads and Jboss worker threads. |
|
I agree on the fact to reduce the impact on users, but since I have the performance hat as well, It's important I remember that what we do for other cases which involve a scarcely allocated resources is not relevant here (UX apart): virtual threads are meant to be allocated and never reused. |
|
I totally follow your point. I've been playing with your benchmarks, I expected that if we eliminate the counter and reuse the threadId the impact would be negligible. |
that would happen only if we do it on logging side only, because logging is not always enabled/on, whilst providing such a composed name is a cost we would pay always, upfront, regardless needed or not. |
|
Honestly, on the logging side, I don't see how we can do this upstream on Jboss logging in a general manner. So again, I think I am on the same position as before. |
got it, so it is more on the jboss logger side; at this point there isn't much we can do, agree. |
I'm not following. What's the problem on the JBoss Logger side? If any, we can certainly suggest improvements and get a quick release. But it's my understanding so far that we should simply be able to change the default pattern, and remove the counter. What am I missing? |
