Higher Score Calculation Speed in Benchmarker

[simontiffert]

On several projects I faced a much higher score calculation speed in the benchmarker compared to the normal mode in Quarkus.

I tried to replicate it with the non-enterprise version 1.13.0 school-timetabling quickstart with the large dataset and could also replicate it locally with multiple runs. Attached are the logs while both runs are done with REPRODUCIBLE, move thread count NONE and parallelBenchmarkCount 1.

1. Start timefold-quickstarts/java/school-timetabling: mvn quarkus:dev
2. Run small dataset in UI (to warm up) - Solving ended: time spent (30002), best score (0hard/11soft), score calculation speed (37199/sec), phase total (2), environment mode (REPRODUCIBLE), move thread count (NONE).
3. Run large dataset in UI - Solving ended: time spent (30000), best score (0hard/-74soft), score calculation speed (29498/sec), phase total (2), environment mode (REPRODUCIBLE), move thread count (NONE).
   Configuration of Quarkus Benchmark with export of large dataset to src/test/resources/large.json:

    <dependency>
      <groupId>ai.timefold.solver</groupId>
      <artifactId>timefold-solver-quarkus-benchmark</artifactId>
    </dependency>

@QuarkusTest
public class BenchmarkTest {
    @Inject
    PlannerBenchmarkFactory benchmarkFactory;

    @Inject
    ObjectMapper objectMapper;

    @Test
    public void benchmark() throws IOException, URISyntaxException {
        Timetable timetable = objectMapper.readValue(new URI("file:src/test/resources/large.json").toURL(), Timetable.class);
        benchmarkFactory.buildPlannerBenchmark(timetable).benchmarkAndShowReportInBrowser();
    }
}

Result for default benchmark run without solverBenchmarkConfig (just commented the test termination in application.properties) - 135738/sec:

2024-09-03 09:59:26,399 INFO  [ai.tim.sol.cor.imp.sol.DefaultSolver] (Timefold-2-BenchmarkThread-1) Solving ended: time spent (30000), best score (0hard/45soft), score calculation speed (135738/sec), phase total (2), environment mode (REPRODUCIBLE), move thread count (NONE).

Any ideas where the much higher score calculation speed is coming from?

quarkus-dev-log.txt
benchmark-log.txt

java -version
openjdk version "21.0.1" 2023-10-17
OpenJDK Runtime Environment (build 21.0.1+12-29)
OpenJDK 64-Bit Server VM (build 21.0.1+12-29, mixed mode, sharing)

Environment: Apple M1 Max 64GB

[triceo]

Hello Simon!

One thing people generally don't know about the Quarkus dev mode is that it actually disables Java's C2 compiler (JIT). This in turn makes Java significantly slower. If the benchmarker had been run outside of Quarkus dev mode (perhaps as a standalone Java application), this magical speed improvement is explained.

If that's not the case though and the benchmarker ran in Quarkus dev mode as well, then this will require a careful investigation. Do let us know.

[simontiffert]

You are correct, as I have started both from the IDE (Quarkus vs. test run) I didn't thought about the environment mode and wasn't aware of disabled JIT in dev mode with this huge speed difference.

 mvn clean package
 java -jar target/quarkus-app/quarkus-run.jar

Solving ended: time spent (30000), best score (0hard/45soft), score calculation speed (138284/sec), phase total (2), environment mode (REPRODUCIBLE), move thread count (NONE).

Thank you very much for the pointer.