Introduce incremental workflow scenarios to GHA

[mhajas]

Closes #354
Closes #355

@kami619 Do we want to add also user scenario within this PR, or we will do it as a follow-up?

[kami619]

I would like to get the user scalability scenario in as well, @mhajas its almost ready in my local.

[kami619]

@ahus1 it would be great if you can review some of the changes we(me and Michal) made for these scenarios. Happy to get on a call on Monday for a walk through as well.

cc: @mhajas I did push the changes we discussed.

PS: Leaving the commit trace as is., if needed we can revert to a previous commit, based on the outcome. I can squash and merge in the end of the review cycle.

[ahus1]

I did a review in another commit. Please review the suggestions, and change/revert them as necessary.

There's also a new FIXME in keycloak-scalability-benchmark.yml as it IMHO misses to run AuthorizationCode with --users-per-realm.

For future reviews, please avoid merging the main branch into the PR, as it makes it harder to review the changes in an IDE.

I found that kcb.sh no longer logged the output to the console, which was a surprise to me and would make it harder for users of the simple runs to analyze their results. Therefore I added 'tee' to have the logs both on the console and in a file.

It might make sense to not log on the console for incremental runs - if you want to do this, feel free to do so.

[kami619]

@ahus1 this change was by design., I can explain :)

[ahus1]

Please do, I'm curious :-)

[kami619]

@ahus1 thanks so much for the review., and sorry about the merge of main into this, it happened accidentally, will be more careful from next time.

I have updated the kcb.sh script to do warm-up by default every time we run incremental, as we would want that anyways, and it probably will keep things simple. tagging @mhajas as this is a change in how we ought to run the incremental runs.

Update the FIXME part in kcb.sh with the duplicated code block to accommodate the right args for AuthorizationCode scenario.

I shall work on adding docs for the --incremental mode and test the new changes and get back.

[kami619]

Here is the example output on the console for an incremental run:

$ ./kcb.sh --scenario=keycloak.scenario.authentication.AuthorizationCode --server-url=https://keycloak-aaa-keycloak.apps.*.example.com --realm-name=test-realm --users-per-sec=10 --increment=10 --measurement=60
INFO: Running benchmark in incremental mode.
INFO: Running warm-up phase.
INFO: Running benchmark with users-per-sec=10, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-10-WARM-UP
INFO: Finished Warm-Up phase, for incremental run.
INFO: Running benchmark with users-per-sec=10, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-10
INFO: Running benchmark with users-per-sec=20, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-20
INFO: Running benchmark with users-per-sec=30, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-30
INFO: Running benchmark with users-per-sec=40, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-40
INFO: Running benchmark with users-per-sec=50, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-50
INFO: Running benchmark with users-per-sec=60, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-60
INFO: Running benchmark with users-per-sec=70, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-70
INFO: Running benchmark with users-per-sec=80, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-80
INFO: Running benchmark with users-per-sec=90, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-90
INFO: Running benchmark with users-per-sec=100, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-100
INFO: Running benchmark with users-per-sec=110, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-110
INFO: Keycloak benchmark failed for users-per-sec=110
INFO: Last Successful workload for scenario keycloak.scenario.authentication.AuthorizationCode is users-per-sec=100.
INFO: Running benchmark with users-per-sec=105, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-105
INFO: Keycloak benchmark was already running for users-per-sec=110 with result 2.
INFO: Keycloak benchmark failed for users-per-sec=110
INFO: Last Successful workload for scenario keycloak.scenario.authentication.AuthorizationCode is users-per-sec=105.
INFO: Running benchmark with users-per-sec=107, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-107
INFO: Running benchmark with users-per-sec=109, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-109
INFO: Running benchmark with users-per-sec=111, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-111
INFO: Running benchmark with users-per-sec=113, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-113
INFO: Running benchmark with users-per-sec=115, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-115
INFO: Running benchmark with users-per-sec=117, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-117
INFO: Running benchmark with users-per-sec=119, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-119
INFO: Keycloak benchmark failed for users-per-sec=119
INFO: Last Successful workload for scenario keycloak.scenario.authentication.AuthorizationCode is users-per-sec=117.
INFO: Running benchmark with users-per-sec=118, result output will be available in: /home/kakella/code/kami619/keycloak-benchmark/benchmark/target/keycloak-benchmark-0.10-SNAPSHOT/bin/../results/incremental-20230612113820/users-per-sec-118
INFO: Keycloak benchmark failed for users-per-sec=118
INFO: Last Successful workload for scenario keycloak.scenario.authentication.AuthorizationCode is users-per-sec=117.
INFO: Reached the limit for scenario keycloak.scenario.authentication.AuthorizationCode with users-per-sec=117.

[ahus1]

I've reviewed the changes and ran the scripts locally. This looks ok to be merged. Once it is merged, I'll run additional tests to see if there is the need for further changes.

[ahus1]

Did some small adjustments, including disabling sticky sessions.

Here are the results of the first run: https://github.com/keycloak/keycloak-benchmark/actions/runs/5255644969/jobs/9495932993