Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add debugger benchmarks #7182

Merged
merged 1 commit into from
Jul 4, 2024
Merged

Add debugger benchmarks #7182

merged 1 commit into from
Jul 4, 2024

Conversation

jpbempel
Copy link
Member

@jpbempel jpbempel commented Jun 12, 2024 •
edited by jira bot
Loading

What Does This Do

benchmarks are run only when a file is modified on paths under
dd-java-agent/agent-debugger/*

Motivation

Benchmarks run for every debugger PR to assess improvement/regression/neutral

Additional Notes

Jira ticket: DEBUG-2440

@pr-commenter
Copy link

pr-commenter bot commented Jun 12, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/benchmarks
git_commit_date 1719942483 1719944214
git_commit_sha 179ce68 e650df8
release_version 1.37.0-SNAPSHOT~179ce68ab6 1.35.0-SNAPSHOT~e650df895a
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1719947283 1719947283
ci_job_id 560225352 560225352
ci_pipeline_id 38179100 38179100
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 48 metrics, 14 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:petclinic:tracing:Remote Config worse
[+22.754µs; +71.568µs] or [+3.374%; +10.612%]		 721.577µs 674.416µs
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.07 s) : 0, 1069760
Total [baseline] (10.434 s) : 0, 10433825
Agent [candidate] (1.072 s) : 0, 1072119
Total [candidate] (10.442 s) : 0, 10441859
section appsec
Agent [baseline] (1.186 s) : 0, 1186201
Total [baseline] (10.535 s) : 0, 10534943
Agent [candidate] (1.182 s) : 0, 1181584
Total [candidate] (10.61 s) : 0, 10609828
section iast
Agent [baseline] (1.178 s) : 0, 1178261
Total [baseline] (10.832 s) : 0, 10831571
Agent [candidate] (1.169 s) : 0, 1169480
Total [candidate] (10.739 s) : 0, 10738864
section profiling
Agent [baseline] (1.271 s) : 0, 1271075
Total [baseline] (10.621 s) : 0, 10621315
Agent [candidate] (1.269 s) : 0, 1268998
Total [candidate] (10.652 s) : 0, 10652409
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.07 s -
Agent appsec 1.186 s 116.441 ms (10.9%)
Agent iast 1.178 s 108.501 ms (10.1%)
Agent profiling 1.271 s 201.315 ms (18.8%)
Total tracing 10.434 s -
Total appsec 10.535 s 101.118 ms (1.0%)
Total iast 10.832 s 397.746 ms (3.8%)
Total profiling 10.621 s 187.49 ms (1.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.072 s -
Agent appsec 1.182 s 109.465 ms (10.2%)
Agent iast 1.169 s 97.361 ms (9.1%)
Agent profiling 1.269 s 196.879 ms (18.4%)
Total tracing 10.442 s -
Total appsec 10.61 s 167.97 ms (1.6%)
Total iast 10.739 s 297.006 ms (2.8%)
Total profiling 10.652 s 210.551 ms (2.0%) 
gantt
    title petclinic - break down per module: candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (669.969 ms) : 0, 669969
BytebuddyAgent [candidate] (662.948 ms) : 0, 662948
GlobalTracer [baseline] (306.623 ms) : 0, 306623
GlobalTracer [candidate] (315.379 ms) : 0, 315379
AppSec [baseline] (50.089 ms) : 0, 50089
AppSec [candidate] (50.514 ms) : 0, 50514
Remote Config [baseline] (674.416 µs) : 0, 674
Remote Config [candidate] (721.577 µs) : 0, 722
Telemetry [baseline] (7.59 ms) : 0, 7590
Telemetry [candidate] (7.639 ms) : 0, 7639
section appsec
BytebuddyAgent [baseline] (679.192 ms) : 0, 679192
BytebuddyAgent [candidate] (676.631 ms) : 0, 676631
GlobalTracer [baseline] (299.218 ms) : 0, 299218
GlobalTracer [candidate] (297.65 ms) : 0, 297650
AppSec [baseline] (153.924 ms) : 0, 153924
AppSec [candidate] (153.706 ms) : 0, 153706
IAST [baseline] (21.539 ms) : 0, 21539
IAST [candidate] (22.548 ms) : 0, 22548
Remote Config [baseline] (634.763 µs) : 0, 635
Remote Config [candidate] (967.575 µs) : 0, 968
Telemetry [baseline] (7.747 ms) : 0, 7747
Telemetry [candidate] (6.901 ms) : 0, 6901
section iast
BytebuddyAgent [baseline] (786.139 ms) : 0, 786139
BytebuddyAgent [candidate] (780.08 ms) : 0, 780080
GlobalTracer [baseline] (296.542 ms) : 0, 296542
GlobalTracer [candidate] (293.661 ms) : 0, 293661
AppSec [baseline] (47.684 ms) : 0, 47684
AppSec [candidate] (46.821 ms) : 0, 46821
IAST [baseline] (26.68 ms) : 0, 26680
IAST [candidate] (26.587 ms) : 0, 26587
Remote Config [baseline] (577.688 µs) : 0, 578
Remote Config [candidate] (592.202 µs) : 0, 592
Telemetry [baseline] (7.081 ms) : 0, 7081
Telemetry [candidate] (8.485 ms) : 0, 8485
section profiling
BytebuddyAgent [baseline] (667.031 ms) : 0, 667031
BytebuddyAgent [candidate] (668.279 ms) : 0, 668279
GlobalTracer [baseline] (390.728 ms) : 0, 390728
GlobalTracer [candidate] (387.682 ms) : 0, 387682
AppSec [baseline] (51.844 ms) : 0, 51844
AppSec [candidate] (51.113 ms) : 0, 51113
Remote Config [baseline] (667.373 µs) : 0, 667
Remote Config [candidate] (779.031 µs) : 0, 779
Telemetry [baseline] (7.364 ms) : 0, 7364
Telemetry [candidate] (7.374 ms) : 0, 7374
ProfilingAgent [baseline] (96.054 ms) : 0, 96054
ProfilingAgent [candidate] (96.752 ms) : 0, 96752
Profiling [baseline] (96.078 ms) : 0, 96078
Profiling [candidate] (96.776 ms) : 0, 96776
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.076 s) : 0, 1075638
Total [baseline] (8.593 s) : 0, 8593222
Agent [candidate] (1.064 s) : 0, 1063743
Total [candidate] (8.58 s) : 0, 8580271
section iast
Agent [baseline] (1.177 s) : 0, 1177113
Total [baseline] (9.006 s) : 0, 9006158
Agent [candidate] (1.17 s) : 0, 1170070
Total [candidate] (9.018 s) : 0, 9017844
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.186 s) : 0, 1186356
Total [baseline] (9.04 s) : 0, 9039589
Agent [candidate] (1.176 s) : 0, 1176343
Total [candidate] (9.016 s) : 0, 9015919
section iast_TELEMETRY_OFF
Agent [baseline] (1.18 s) : 0, 1179755
Total [baseline] (9.019 s) : 0, 9018660
Agent [candidate] (1.167 s) : 0, 1167344
Total [candidate] (9.019 s) : 0, 9019491
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.076 s -
Agent iast 1.177 s 101.475 ms (9.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.186 s 110.718 ms (10.3%)
Agent iast_TELEMETRY_OFF 1.18 s 104.117 ms (9.7%)
Total tracing 8.593 s -
Total iast 9.006 s 412.936 ms (4.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.04 s 446.367 ms (5.2%)
Total iast_TELEMETRY_OFF 9.019 s 425.439 ms (5.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.064 s -
Agent iast 1.17 s 106.328 ms (10.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.176 s 112.601 ms (10.6%)
Agent iast_TELEMETRY_OFF 1.167 s 103.601 ms (9.7%)
Total tracing 8.58 s -
Total iast 9.018 s 437.573 ms (5.1%)
Total iast_HARDCODED_SECRET_DISABLED 9.016 s 435.648 ms (5.1%)
Total iast_TELEMETRY_OFF 9.019 s 439.22 ms (5.1%) 
gantt
    title insecure-bank - break down per module: candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (674.025 ms) : 0, 674025
BytebuddyAgent [candidate] (655.505 ms) : 0, 655505
GlobalTracer [baseline] (307.671 ms) : 0, 307671
GlobalTracer [candidate] (314.909 ms) : 0, 314909
AppSec [baseline] (50.621 ms) : 0, 50621
AppSec [candidate] (50.479 ms) : 0, 50479
Remote Config [baseline] (750.41 µs) : 0, 750
Remote Config [candidate] (711.662 µs) : 0, 712
Telemetry [baseline] (7.688 ms) : 0, 7688
Telemetry [candidate] (7.67 ms) : 0, 7670
section iast
BytebuddyAgent [baseline] (784.154 ms) : 0, 784154
BytebuddyAgent [candidate] (780.357 ms) : 0, 780357
GlobalTracer [baseline] (296.443 ms) : 0, 296443
GlobalTracer [candidate] (293.867 ms) : 0, 293867
AppSec [baseline] (47.359 ms) : 0, 47359
AppSec [candidate] (46.486 ms) : 0, 46486
IAST [baseline] (28.129 ms) : 0, 28129
IAST [candidate] (27.87 ms) : 0, 27870
Remote Config [baseline] (561.942 µs) : 0, 562
Remote Config [candidate] (583.079 µs) : 0, 583
Telemetry [baseline] (6.972 ms) : 0, 6972
Telemetry [candidate] (7.613 ms) : 0, 7613
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (790.494 ms) : 0, 790494
BytebuddyAgent [candidate] (784.793 ms) : 0, 784793
GlobalTracer [baseline] (298.495 ms) : 0, 298495
GlobalTracer [candidate] (295.552 ms) : 0, 295552
AppSec [baseline] (47.854 ms) : 0, 47854
AppSec [candidate] (46.885 ms) : 0, 46885
IAST [baseline] (27.423 ms) : 0, 27423
IAST [candidate] (26.702 ms) : 0, 26702
Remote Config [baseline] (600.908 µs) : 0, 601
Remote Config [candidate] (586.646 µs) : 0, 587
Telemetry [baseline] (7.837 ms) : 0, 7837
Telemetry [candidate] (8.366 ms) : 0, 8366
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (785.99 ms) : 0, 785990
BytebuddyAgent [candidate] (779.735 ms) : 0, 779735
GlobalTracer [baseline] (297.331 ms) : 0, 297331
GlobalTracer [candidate] (294.308 ms) : 0, 294308
AppSec [baseline] (47.537 ms) : 0, 47537
AppSec [candidate] (51.326 ms) : 0, 51326
IAST [baseline] (27.64 ms) : 0, 27640
IAST [candidate] (21.16 ms) : 0, 21160
Remote Config [baseline] (631.033 µs) : 0, 631
Remote Config [candidate] (582.117 µs) : 0, 582
Telemetry [baseline] (6.984 ms) : 0, 6984
Telemetry [candidate] (6.87 ms) : 0, 6870
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-07-02T18:38:40 2024-07-02T18:45:28
git_branch master jpbempel/benchmarks
git_commit_date 1719942483 1719944214
git_commit_sha 179ce68 e650df8
release_version 1.37.0-SNAPSHOT~179ce68ab6 1.35.0-SNAPSHOT~e650df895a
start_time 2024-07-02T18:38:27 2024-07-02T18:45:15
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1719946271 1719946271
ci_job_id 560225353 560225353
ci_pipeline_id 38179100 38179100
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.343 ms) : 1325, 1362
.   : milestone, 1343,
appsec (1.708 ms) : 1684, 1731
.   : milestone, 1708,
appsec_no_iast (1.726 ms) : 1702, 1750
.   : milestone, 1726,
iast (1.47 ms) : 1447, 1493
.   : milestone, 1470,
profiling (1.487 ms) : 1463, 1512
.   : milestone, 1487,
tracing (1.475 ms) : 1450, 1499
.   : milestone, 1475,
section candidate
no_agent (1.348 ms) : 1329, 1367
.   : milestone, 1348,
appsec (1.735 ms) : 1711, 1759
.   : milestone, 1735,
appsec_no_iast (1.692 ms) : 1667, 1717
.   : milestone, 1692,
iast (1.486 ms) : 1462, 1510
.   : milestone, 1486,
profiling (1.488 ms) : 1463, 1512
.   : milestone, 1488,
tracing (1.459 ms) : 1435, 1484
.   : milestone, 1459,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.343 ms [1.325 ms, 1.362 ms] -
appsec 1.708 ms [1.684 ms, 1.731 ms] 364.292 µs (27.1%)
appsec_no_iast 1.726 ms [1.702 ms, 1.75 ms] 382.687 µs (28.5%)
iast 1.47 ms [1.447 ms, 1.493 ms] 126.432 µs (9.4%)
profiling 1.487 ms [1.463 ms, 1.512 ms] 143.647 µs (10.7%)
tracing 1.475 ms [1.45 ms, 1.499 ms] 131.235 µs (9.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.348 ms [1.329 ms, 1.367 ms] -
appsec 1.735 ms [1.711 ms, 1.759 ms] 387.053 µs (28.7%)
appsec_no_iast 1.692 ms [1.667 ms, 1.717 ms] 343.593 µs (25.5%)
iast 1.486 ms [1.462 ms, 1.51 ms] 137.817 µs (10.2%)
profiling 1.488 ms [1.463 ms, 1.512 ms] 139.594 µs (10.4%)
tracing 1.459 ms [1.435 ms, 1.484 ms] 110.914 µs (8.2%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.384 µs) : 348, 387
.   : milestone, 367,
iast (474.317 µs) : 453, 496
.   : milestone, 474,
iast_FULL (548.218 µs) : 527, 569
.   : milestone, 548,
iast_GLOBAL (509.186 µs) : 487, 531
.   : milestone, 509,
iast_HARDCODED_SECRET_DISABLED (476.485 µs) : 456, 497
.   : milestone, 476,
iast_INACTIVE (452.448 µs) : 430, 474
.   : milestone, 452,
iast_TELEMETRY_OFF (466.463 µs) : 446, 487
.   : milestone, 466,
tracing (447.387 µs) : 426, 468
.   : milestone, 447,
section candidate
no_agent (368.761 µs) : 348, 389
.   : milestone, 369,
iast (476.246 µs) : 455, 497
.   : milestone, 476,
iast_FULL (545.913 µs) : 525, 567
.   : milestone, 546,
iast_GLOBAL (502.543 µs) : 481, 524
.   : milestone, 503,
iast_HARDCODED_SECRET_DISABLED (478.54 µs) : 458, 499
.   : milestone, 479,
iast_INACTIVE (454.297 µs) : 432, 476
.   : milestone, 454,
iast_TELEMETRY_OFF (471.834 µs) : 450, 493
.   : milestone, 472,
tracing (439.729 µs) : 419, 460
.   : milestone, 440,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 367.384 µs [348.122 µs, 386.647 µs] -
iast 474.317 µs [453.055 µs, 495.579 µs] 106.933 µs (29.1%)
iast_FULL 548.218 µs [526.982 µs, 569.454 µs] 180.834 µs (49.2%)
iast_GLOBAL 509.186 µs [487.175 µs, 531.198 µs] 141.802 µs (38.6%)
iast_HARDCODED_SECRET_DISABLED 476.485 µs [455.526 µs, 497.444 µs] 109.101 µs (29.7%)
iast_INACTIVE 452.448 µs [430.469 µs, 474.427 µs] 85.064 µs (23.2%)
iast_TELEMETRY_OFF 466.463 µs [445.794 µs, 487.132 µs] 99.078 µs (27.0%)
tracing 447.387 µs [426.469 µs, 468.305 µs] 80.003 µs (21.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.761 µs [348.178 µs, 389.344 µs] -
iast 476.246 µs [455.32 µs, 497.171 µs] 107.485 µs (29.1%)
iast_FULL 545.913 µs [524.616 µs, 567.21 µs] 177.152 µs (48.0%)
iast_GLOBAL 502.543 µs [481.098 µs, 523.989 µs] 133.782 µs (36.3%)
iast_HARDCODED_SECRET_DISABLED 478.54 µs [457.597 µs, 499.483 µs] 109.779 µs (29.8%)
iast_INACTIVE 454.297 µs [432.464 µs, 476.131 µs] 85.536 µs (23.2%)
iast_TELEMETRY_OFF 471.834 µs [450.441 µs, 493.226 µs] 103.072 µs (28.0%)
tracing 439.729 µs [419.207 µs, 460.25 µs] 70.967 µs (19.2%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/benchmarks
git_commit_date 1719942483 1719944214
git_commit_sha 179ce68 e650df8
release_version 1.37.0-SNAPSHOT~179ce68ab6 1.35.0-SNAPSHOT~e650df895a
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1719946664 1719946664
ci_job_id 560225354 560225354
ci_pipeline_id 38179100 38179100
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.464 ms) : 1453, 1476
.   : milestone, 1464,
appsec (2.25 ms) : 2213, 2286
.   : milestone, 2250,
iast (1.987 ms) : 1943, 2030
.   : milestone, 1987,
iast_GLOBAL (2.037 ms) : 1992, 2081
.   : milestone, 2037,
profiling (1.879 ms) : 1844, 1914
.   : milestone, 1879,
tracing (1.855 ms) : 1821, 1890
.   : milestone, 1855,
section candidate
no_agent (1.471 ms) : 1459, 1483
.   : milestone, 1471,
appsec (2.209 ms) : 2175, 2243
.   : milestone, 2209,
iast (1.965 ms) : 1923, 2006
.   : milestone, 1965,
iast_GLOBAL (2.005 ms) : 1963, 2046
.   : milestone, 2005,
profiling (1.844 ms) : 1811, 1876
.   : milestone, 1844,
tracing (1.841 ms) : 1809, 1874
.   : milestone, 1841,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.464 ms [1.453 ms, 1.476 ms] -
appsec 2.25 ms [2.213 ms, 2.286 ms] 785.736 µs (53.7%)
iast 1.987 ms [1.943 ms, 2.03 ms] 522.336 µs (35.7%)
iast_GLOBAL 2.037 ms [1.992 ms, 2.081 ms] 572.417 µs (39.1%)
profiling 1.879 ms [1.844 ms, 1.914 ms] 415.22 µs (28.4%)
tracing 1.855 ms [1.821 ms, 1.89 ms] 391.201 µs (26.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.471 ms [1.459 ms, 1.483 ms] -
appsec 2.209 ms [2.175 ms, 2.243 ms] 737.861 µs (50.2%)
iast 1.965 ms [1.923 ms, 2.006 ms] 493.527 µs (33.5%)
iast_GLOBAL 2.005 ms [1.963 ms, 2.046 ms] 533.489 µs (36.3%)
profiling 1.844 ms [1.811 ms, 1.876 ms] 372.506 µs (25.3%)
tracing 1.841 ms [1.809 ms, 1.874 ms] 370.132 µs (25.2%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.717 s) : 14717000, 14717000
.   : milestone, 14717000,
appsec (15.106 s) : 15106000, 15106000
.   : milestone, 15106000,
iast (18.841 s) : 18841000, 18841000
.   : milestone, 18841000,
iast_GLOBAL (17.857 s) : 17857000, 17857000
.   : milestone, 17857000,
profiling (15.365 s) : 15365000, 15365000
.   : milestone, 15365000,
tracing (14.871 s) : 14871000, 14871000
.   : milestone, 14871000,
section candidate
no_agent (15.37 s) : 15370000, 15370000
.   : milestone, 15370000,
appsec (15.056 s) : 15056000, 15056000
.   : milestone, 15056000,
iast (19.008 s) : 19008000, 19008000
.   : milestone, 19008000,
iast_GLOBAL (17.817 s) : 17817000, 17817000
.   : milestone, 17817000,
profiling (14.978 s) : 14978000, 14978000
.   : milestone, 14978000,
tracing (15.106 s) : 15106000, 15106000
.   : milestone, 15106000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.717 s [14.717 s, 14.717 s] -
appsec 15.106 s [15.106 s, 15.106 s] 389.0 ms (2.6%)
iast 18.841 s [18.841 s, 18.841 s] 4.124 s (28.0%)
iast_GLOBAL 17.857 s [17.857 s, 17.857 s] 3.14 s (21.3%)
profiling 15.365 s [15.365 s, 15.365 s] 648.0 ms (4.4%)
tracing 14.871 s [14.871 s, 14.871 s] 154.0 ms (1.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.37 s [15.37 s, 15.37 s] -
appsec 15.056 s [15.056 s, 15.056 s] -314.0 ms (-2.0%)
iast 19.008 s [19.008 s, 19.008 s] 3.638 s (23.7%)
iast_GLOBAL 17.817 s [17.817 s, 17.817 s] 2.447 s (15.9%)
profiling 14.978 s [14.978 s, 14.978 s] -392.0 ms (-2.6%)
tracing 15.106 s [15.106 s, 15.106 s] -264.0 ms (-1.7%)

@jpbempel jpbempel force-pushed the jpbempel/benchmarks branch 2 times, most recently from b0d0e89 to 8b94822 Compare June 13, 2024 07:27
@pr-commenter
Copy link

pr-commenter bot commented Jun 13, 2024 •
edited
Loading

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1719946019 1719946437
end_time 2024-07-02T18:48:14 2024-07-02T18:55:11
git_branch master jpbempel/benchmarks
git_commit_sha 179ce68 e650df8
start_time 2024-07-02T18:47:00 2024-07-02T18:53:58
See matching parameters
Baseline Candidate
ci_job_id 560225358 560225358
ci_pipeline_id 38179100 38179100
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date 1719944214 1719944214

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 7 unstable metrics.

See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-45.555µs; +33.979µs] or [-16.915%; +12.617%]		 unstable
[-65.572µs; +41.477µs] or [-21.212%; +13.417%]		 unstable
[-79.631µs; +50.118µs] or [-24.508%; +15.425%]		 unstable
[-840.459µs; +272.644µs] or [-92.034%; +29.856%]		 unstable
[-97.762op/s; +169.861op/s] or [-3.666%; +6.370%]
scenario:basic same same same unstable
[-42499.753ns; +43018.180ns] or [-6.922%; +7.006%]		 unstable
[-155.438op/s; +155.438op/s] or [-6.218%; +6.218%]
scenario:loop unsure
[-53.540µs; -30.781µs] or [-0.439%; -0.253%]		 unsure
[-103.275µs; -32.754µs] or [-0.833%; -0.264%]		 unsure
[-177.760µs; -12.106µs] or [-1.415%; -0.096%]		 same same
Request duration reports for reports 
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (309.134 µs) : 247, 372
.   : milestone, 309,
basic (312.871 µs) : 299, 326
.   : milestone, 313,
loop (12.405 ms) : 12374, 12436
.   : milestone, 12405,
section candidate
noprobe (297.086 µs) : 261, 333
.   : milestone, 297,
basic (314.752 µs) : 305, 325
.   : milestone, 315,
loop (12.337 ms) : 12303, 12372
.   : milestone, 12337,
Loading
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 309.134 µs [246.535 µs, 371.732 µs]
basic 312.871 µs [299.368 µs, 326.373 µs]
loop 12.405 ms [12.374 ms, 12.436 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 297.086 µs [261.092 µs, 333.08 µs]
basic 314.752 µs [304.574 µs, 324.93 µs]
loop 12.337 ms [12.303 ms, 12.372 ms]

@jpbempel jpbempel force-pushed the jpbempel/benchmarks branch 12 times, most recently from 9bba04a to c2f4273 Compare June 21, 2024 18:49
@jpbempel
Add debugger benchmarks
e650df8 
benchmarks are run only when a file is modified on paths under
dd-java-agent/agent-debugger/*
@jpbempel jpbempel changed the title DRAFT Add debugger benchmarks Jul 2, 2024
@jpbempel jpbempel marked this pull request as ready for review July 2, 2024 19:01
@jpbempel jpbempel requested a review from a team as a code owner July 2, 2024 19:01
@jpbempel jpbempel added the comp: debugger Dynamic Instrumentation label Jul 3, 2024
@PerfectSlayer PerfectSlayer added comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes labels Jul 4, 2024
@jpbempel jpbempel merged commit 0327c4d into master Jul 4, 2024
82 checks passed
@jpbempel jpbempel deleted the jpbempel/benchmarks branch July 4, 2024 14:38
@github-actions github-actions bot added this to the 1.38.0 milestone Jul 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes

@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: debugger Dynamic Instrumentation comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
1.38.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@jpbempel @PerfectSlayer