Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mark Spring Webflux 5 integration test as flaky #6917

Merged
merged 1 commit into from
Apr 16, 2024
Merged

Mark Spring Webflux 5 integration test as flaky #6917

merged 1 commit into from
Apr 16, 2024

Conversation

PerfectSlayer
Copy link
Contributor

@PerfectSlayer PerfectSlayer commented Apr 16, 2024 •
edited by jira bot
Loading

What Does This Do

This PR marks Spring Webflux 5 integration test as flaky.

Motivation

See #6909

Additional Notes

Jira ticket: AIDM-11

@PerfectSlayer PerfectSlayer added tag: flaky test Flaky tests inst: spring Spring instrumentation tag: flaky test/disabled Disabled flaky tests labels Apr 16, 2024
@PerfectSlayer PerfectSlayer requested a review from a team as a code owner April 16, 2024 08:01
@PerfectSlayer PerfectSlayer added the tag: no release notes Changes to exclude from release notes label Apr 16, 2024
@pr-commenter
Copy link

pr-commenter bot commented Apr 16, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master bbujon/ci-flaky-integration-spring-webflux
git_commit_date 1713248465 1713254341
git_commit_sha 4c97fc1 7ec3c80
release_version 1.33.0-SNAPSHOT~4c97fc1a28 1.33.0-SNAPSHOT~7ec3c800b6
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1713257743 1713257743
ci_job_id 487467829 487467829
ci_pipeline_id 32170573 32170573
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 15 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.33.0-SNAPSHOT~7ec3c800b6, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.09 s) : 0, 1089896
Total [baseline] (8.564 s) : 0, 8563923
Agent [candidate] (1.077 s) : 0, 1077210
Total [candidate] (8.555 s) : 0, 8555339
section iast
Agent [baseline] (1.199 s) : 0, 1199322
Total [baseline] (9.057 s) : 0, 9056648
Agent [candidate] (1.199 s) : 0, 1199087
Total [candidate] (9.006 s) : 0, 9006260
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.209 s) : 0, 1208677
Total [baseline] (9.03 s) : 0, 9029819
Agent [candidate] (1.201 s) : 0, 1200848
Total [candidate] (8.994 s) : 0, 8993983
section iast_TELEMETRY_OFF
Agent [baseline] (1.196 s) : 0, 1195922
Total [baseline] (8.999 s) : 0, 8998918
Agent [candidate] (1.197 s) : 0, 1197395
Total [candidate] (9.008 s) : 0, 9008436
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.09 s -
Agent iast 1.199 s 109.426 ms (10.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.209 s 118.781 ms (10.9%)
Agent iast_TELEMETRY_OFF 1.196 s 106.026 ms (9.7%)
Total tracing 8.564 s -
Total iast 9.057 s 492.725 ms (5.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.03 s 465.896 ms (5.4%)
Total iast_TELEMETRY_OFF 8.999 s 434.996 ms (5.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.077 s -
Agent iast 1.199 s 121.877 ms (11.3%)
Agent iast_HARDCODED_SECRET_DISABLED 1.201 s 123.638 ms (11.5%)
Agent iast_TELEMETRY_OFF 1.197 s 120.185 ms (11.2%)
Total tracing 8.555 s -
Total iast 9.006 s 450.921 ms (5.3%)
Total iast_HARDCODED_SECRET_DISABLED 8.994 s 438.643 ms (5.1%)
Total iast_TELEMETRY_OFF 9.008 s 453.097 ms (5.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.33.0-SNAPSHOT~7ec3c800b6, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (684.578 ms) : 0, 684578
BytebuddyAgent [candidate] (674.762 ms) : 0, 674762
GlobalTracer [baseline] (312.143 ms) : 0, 312143
GlobalTracer [candidate] (310.088 ms) : 0, 310088
AppSec [baseline] (49.919 ms) : 0, 49919
AppSec [candidate] (49.629 ms) : 0, 49629
Remote Config [baseline] (668.716 µs) : 0, 669
Remote Config [candidate] (656.088 µs) : 0, 656
Telemetry [baseline] (7.674 ms) : 0, 7674
Telemetry [candidate] (7.573 ms) : 0, 7573
section iast
BytebuddyAgent [baseline] (794.694 ms) : 0, 794694
BytebuddyAgent [candidate] (794.53 ms) : 0, 794530
GlobalTracer [baseline] (288.338 ms) : 0, 288338
GlobalTracer [candidate] (288.478 ms) : 0, 288478
AppSec [baseline] (50.286 ms) : 0, 50286
AppSec [candidate] (50.312 ms) : 0, 50312
IAST [baseline] (23.672 ms) : 0, 23672
IAST [candidate] (24.288 ms) : 0, 24288
Remote Config [baseline] (574.295 µs) : 0, 574
Remote Config [candidate] (592.022 µs) : 0, 592
Telemetry [baseline] (7.336 ms) : 0, 7336
Telemetry [candidate] (6.6 ms) : 0, 6600
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (801.003 ms) : 0, 801003
BytebuddyAgent [candidate] (795.638 ms) : 0, 795638
GlobalTracer [baseline] (290.529 ms) : 0, 290529
GlobalTracer [candidate] (289.277 ms) : 0, 289277
AppSec [baseline] (50.508 ms) : 0, 50508
AppSec [candidate] (47.34 ms) : 0, 47340
IAST [baseline] (23.939 ms) : 0, 23939
IAST [candidate] (26.105 ms) : 0, 26105
Remote Config [baseline] (607.828 µs) : 0, 608
Remote Config [candidate] (583.33 µs) : 0, 583
Telemetry [baseline] (7.482 ms) : 0, 7482
Telemetry [candidate] (7.389 ms) : 0, 7389
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (791.372 ms) : 0, 791372
BytebuddyAgent [candidate] (792.385 ms) : 0, 792385
GlobalTracer [baseline] (287.765 ms) : 0, 287765
GlobalTracer [candidate] (289.1 ms) : 0, 289100
AppSec [baseline] (47.721 ms) : 0, 47721
AppSec [candidate] (51.074 ms) : 0, 51074
IAST [baseline] (27.478 ms) : 0, 27478
IAST [candidate] (22.537 ms) : 0, 22537
Remote Config [baseline] (586.913 µs) : 0, 587
Remote Config [candidate] (590.787 µs) : 0, 591
Telemetry [baseline] (6.587 ms) : 0, 6587
Telemetry [candidate] (7.373 ms) : 0, 7373
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.33.0-SNAPSHOT~7ec3c800b6, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.092 s) : 0, 1091950
Total [baseline] (10.439 s) : 0, 10439252
Agent [candidate] (1.084 s) : 0, 1084469
Total [candidate] (10.403 s) : 0, 10402903
section appsec
Agent [baseline] (1.202 s) : 0, 1202275
Total [baseline] (10.537 s) : 0, 10537137
Agent [candidate] (1.224 s) : 0, 1223699
Total [candidate] (10.516 s) : 0, 10515940
section iast
Agent [baseline] (1.198 s) : 0, 1197518
Total [baseline] (10.688 s) : 0, 10687914
Agent [candidate] (1.199 s) : 0, 1198981
Total [candidate] (10.693 s) : 0, 10692681
section profiling
Agent [baseline] (1.269 s) : 0, 1269273
Total [baseline] (10.613 s) : 0, 10613207
Agent [candidate] (1.27 s) : 0, 1269687
Total [candidate] (10.601 s) : 0, 10601430
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.092 s -
Agent appsec 1.202 s 110.326 ms (10.1%)
Agent iast 1.198 s 105.568 ms (9.7%)
Agent profiling 1.269 s 177.323 ms (16.2%)
Total tracing 10.439 s -
Total appsec 10.537 s 97.885 ms (0.9%)
Total iast 10.688 s 248.663 ms (2.4%)
Total profiling 10.613 s 173.955 ms (1.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.084 s -
Agent appsec 1.224 s 139.23 ms (12.8%)
Agent iast 1.199 s 114.512 ms (10.6%)
Agent profiling 1.27 s 185.218 ms (17.1%)
Total tracing 10.403 s -
Total appsec 10.516 s 113.037 ms (1.1%)
Total iast 10.693 s 289.778 ms (2.8%)
Total profiling 10.601 s 198.527 ms (1.9%) 
gantt
    title petclinic - break down per module: candidate=1.33.0-SNAPSHOT~7ec3c800b6, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (684.331 ms) : 0, 684331
BytebuddyAgent [candidate] (679.145 ms) : 0, 679145
GlobalTracer [baseline] (314.149 ms) : 0, 314149
GlobalTracer [candidate] (312.466 ms) : 0, 312466
AppSec [baseline] (50.12 ms) : 0, 50120
AppSec [candidate] (49.891 ms) : 0, 49891
Remote Config [baseline] (678.128 µs) : 0, 678
Remote Config [candidate] (662.767 µs) : 0, 663
Telemetry [baseline] (7.725 ms) : 0, 7725
Telemetry [candidate] (7.602 ms) : 0, 7602
section appsec
BytebuddyAgent [baseline] (697.201 ms) : 0, 697201
BytebuddyAgent [candidate] (710.11 ms) : 0, 710110
GlobalTracer [baseline] (293.641 ms) : 0, 293641
GlobalTracer [candidate] (297.981 ms) : 0, 297981
AppSec [baseline] (150.399 ms) : 0, 150399
AppSec [candidate] (151.717 ms) : 0, 151717
IAST [baseline] (19.034 ms) : 0, 19034
IAST [candidate] (19.31 ms) : 0, 19310
Remote Config [baseline] (617.956 µs) : 0, 618
Remote Config [candidate] (623.646 µs) : 0, 624
Telemetry [baseline] (6.867 ms) : 0, 6867
Telemetry [candidate] (8.879 ms) : 0, 8879
section iast
BytebuddyAgent [baseline] (793.392 ms) : 0, 793392
BytebuddyAgent [candidate] (794.6 ms) : 0, 794600
GlobalTracer [baseline] (287.832 ms) : 0, 287832
GlobalTracer [candidate] (288.424 ms) : 0, 288424
AppSec [baseline] (50.048 ms) : 0, 50048
AppSec [candidate] (49.303 ms) : 0, 49303
IAST [baseline] (23.139 ms) : 0, 23139
IAST [candidate] (24.426 ms) : 0, 24426
Remote Config [baseline] (576.962 µs) : 0, 577
Remote Config [candidate] (574.177 µs) : 0, 574
Telemetry [baseline] (8.129 ms) : 0, 8129
Telemetry [candidate] (7.374 ms) : 0, 7374
section profiling
BytebuddyAgent [baseline] (677.369 ms) : 0, 677369
BytebuddyAgent [candidate] (678.467 ms) : 0, 678467
GlobalTracer [baseline] (379.96 ms) : 0, 379960
GlobalTracer [candidate] (380.763 ms) : 0, 380763
AppSec [baseline] (50.506 ms) : 0, 50506
AppSec [candidate] (50.383 ms) : 0, 50383
Remote Config [baseline] (721.096 µs) : 0, 721
Remote Config [candidate] (706.027 µs) : 0, 706
Telemetry [baseline] (7.439 ms) : 0, 7439
Telemetry [candidate] (7.455 ms) : 0, 7455
ProfilingAgent [baseline] (96.901 ms) : 0, 96901
ProfilingAgent [candidate] (95.502 ms) : 0, 95502
Profiling [baseline] (96.925 ms) : 0, 96925
Profiling [candidate] (95.525 ms) : 0, 95525
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-04-16T08:27:56 2024-04-16T08:49:54
git_branch master bbujon/ci-flaky-integration-spring-webflux
git_commit_date 1713248465 1713254341
git_commit_sha 4c97fc1 7ec3c80
release_version 1.33.0-SNAPSHOT~4c97fc1a28 1.33.0-SNAPSHOT~7ec3c800b6
start_time 2024-04-16T08:27:42 2024-04-16T08:49:41
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1713257743 1713257743
ci_job_id 487467829 487467829
ci_pipeline_id 32170573 32170573
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.33.0-SNAPSHOT~7ec3c800b6, baseline=1.33.0-SNAPSHOT~4c97fc1a28
    dateFormat X
    axisFormat %s
section baseline
no_agent (372.587 µs) : 353, 392
.   : milestone, 373,
iast (480.772 µs) : 459, 502
.   : milestone, 481,
iast_FULL (542.632 µs) : 521, 564
.   : milestone, 543,
iast_GLOBAL (511.537 µs) : 490, 533
.   : milestone, 512,
iast_HARDCODED_SECRET_DISABLED (477.337 µs) : 456, 498
.   : milestone, 477,
iast_INACTIVE (447.518 µs) : 427, 468
.   : milestone, 448,
iast_TELEMETRY_OFF (472.747 µs) : 452, 494
.   : milestone, 473,
tracing (446.307 µs) : 426, 467
.   : milestone, 446,
section candidate
no_agent (370.812 µs) : 351, 390
.   : milestone, 371,
iast (480.561 µs) : 459, 502
.   : milestone, 481,
iast_FULL (540.284 µs) : 519, 561
.   : milestone, 540,
iast_GLOBAL (512.473 µs) : 490, 535
.   : milestone, 512,
iast_HARDCODED_SECRET_DISABLED (476.106 µs) : 455, 497
.   : milestone, 476,
iast_INACTIVE (450.443 µs) : 429, 472
.   : milestone, 450,
iast_TELEMETRY_OFF (469.586 µs) : 449, 490
.   : milestone, 470,
tracing (442.235 µs) : 422, 463
.   : milestone, 442,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 372.587 µs [353.079 µs, 392.094 µs] -
iast 480.772 µs [459.36 µs, 502.183 µs] 108.185 µs (29.0%)
iast_FULL 542.632 µs [521.367 µs, 563.897 µs] 170.045 µs (45.6%)
iast_GLOBAL 511.537 µs [489.81 µs, 533.264 µs] 138.95 µs (37.3%)
iast_HARDCODED_SECRET_DISABLED 477.337 µs [456.333 µs, 498.342 µs] 104.75 µs (28.1%)
iast_INACTIVE 447.518 µs [427.092 µs, 467.945 µs] 74.931 µs (20.1%)
iast_TELEMETRY_OFF 472.747 µs [451.895 µs, 493.599 µs] 100.16 µs (26.9%)
tracing 446.307 µs [425.941 µs, 466.673 µs] 73.721 µs (19.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 370.812 µs [351.196 µs, 390.427 µs] -
iast 480.561 µs [459.036 µs, 502.086 µs] 109.749 µs (29.6%)
iast_FULL 540.284 µs [519.184 µs, 561.384 µs] 169.472 µs (45.7%)
iast_GLOBAL 512.473 µs [490.267 µs, 534.678 µs] 141.661 µs (38.2%)
iast_HARDCODED_SECRET_DISABLED 476.106 µs [455.103 µs, 497.11 µs] 105.295 µs (28.4%)
iast_INACTIVE 450.443 µs [428.96 µs, 471.925 µs] 79.631 µs (21.5%)
iast_TELEMETRY_OFF 469.586 µs [448.928 µs, 490.244 µs] 98.774 µs (26.6%)
tracing 442.235 µs [421.877 µs, 462.592 µs] 71.423 µs (19.3%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.33.0-SNAPSHOT~7ec3c800b6, baseline=1.33.0-SNAPSHOT~4c97fc1a28
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.339 ms) : 1320, 1359
.   : milestone, 1339,
appsec (1.719 ms) : 1694, 1743
.   : milestone, 1719,
appsec_no_iast (1.707 ms) : 1682, 1732
.   : milestone, 1707,
iast (1.488 ms) : 1465, 1511
.   : milestone, 1488,
profiling (1.529 ms) : 1502, 1555
.   : milestone, 1529,
tracing (1.498 ms) : 1474, 1523
.   : milestone, 1498,
section candidate
no_agent (1.348 ms) : 1330, 1367
.   : milestone, 1348,
appsec (1.75 ms) : 1727, 1774
.   : milestone, 1750,
appsec_no_iast (1.752 ms) : 1728, 1776
.   : milestone, 1752,
iast (1.494 ms) : 1471, 1517
.   : milestone, 1494,
profiling (1.499 ms) : 1473, 1524
.   : milestone, 1499,
tracing (1.519 ms) : 1494, 1543
.   : milestone, 1519,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.339 ms [1.32 ms, 1.359 ms] -
appsec 1.719 ms [1.694 ms, 1.743 ms] 379.476 µs (28.3%)
appsec_no_iast 1.707 ms [1.682 ms, 1.732 ms] 367.74 µs (27.5%)
iast 1.488 ms [1.465 ms, 1.511 ms] 148.977 µs (11.1%)
profiling 1.529 ms [1.502 ms, 1.555 ms] 189.479 µs (14.1%)
tracing 1.498 ms [1.474 ms, 1.523 ms] 159.077 µs (11.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.348 ms [1.33 ms, 1.367 ms] -
appsec 1.75 ms [1.727 ms, 1.774 ms] 401.643 µs (29.8%)
appsec_no_iast 1.752 ms [1.728 ms, 1.776 ms] 403.56 µs (29.9%)
iast 1.494 ms [1.471 ms, 1.517 ms] 145.396 µs (10.8%)
profiling 1.499 ms [1.473 ms, 1.524 ms] 150.265 µs (11.1%)
tracing 1.519 ms [1.494 ms, 1.543 ms] 170.124 µs (12.6%)

@PerfectSlayer PerfectSlayer merged commit ffbd067 into master Apr 16, 2024
79 checks passed
@PerfectSlayer PerfectSlayer deleted the bbujon/ci-flaky-integration-spring-webflux branch April 16, 2024 17:37
@github-actions github-actions bot added this to the 1.33.0 milestone Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amarziali amarziali amarziali approved these changes

@ygree ygree Awaiting requested review from ygree ygree was automatically assigned from DataDog/apm-java

@am312 am312 Awaiting requested review from am312 am312 was automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
inst: spring Spring instrumentation tag: flaky test/disabled Disabled flaky tests tag: flaky test Flaky tests tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
1.33.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@PerfectSlayer @amarziali