-
Notifications
You must be signed in to change notification settings - Fork 279
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Track JVM RSS in JDK 21+ #7227
Track JVM RSS in JDK 21+ #7227
Conversation
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 1 performance regressions! Performance is the same for 50 metrics, 12 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.36.0-SNAPSHOT~d2672ea954, baseline=1.38.0-SNAPSHOT~120b584fde
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.073 s) : 0, 1072873
Total [baseline] (8.518 s) : 0, 8518172
Agent [candidate] (1.059 s) : 0, 1058919
Total [candidate] (8.502 s) : 0, 8501768
section iast
Agent [baseline] (1.169 s) : 0, 1168982
Total [baseline] (8.98 s) : 0, 8980220
Agent [candidate] (1.168 s) : 0, 1167679
Total [candidate] (8.977 s) : 0, 8977133
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.17 s) : 0, 1169607
Total [baseline] (8.93 s) : 0, 8930420
Agent [candidate] (1.173 s) : 0, 1173260
Total [candidate] (8.967 s) : 0, 8967138
section iast_TELEMETRY_OFF
Agent [baseline] (1.168 s) : 0, 1168116
Total [baseline] (8.948 s) : 0, 8947741
Agent [candidate] (1.167 s) : 0, 1166612
Total [candidate] (8.947 s) : 0, 8946538
gantt
title insecure-bank - break down per module: candidate=1.36.0-SNAPSHOT~d2672ea954, baseline=1.38.0-SNAPSHOT~120b584fde
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (673.706 ms) : 0, 673706
BytebuddyAgent [candidate] (662.825 ms) : 0, 662825
GlobalTracer [baseline] (305.759 ms) : 0, 305759
GlobalTracer [candidate] (303.54 ms) : 0, 303540
AppSec [baseline] (50.03 ms) : 0, 50030
AppSec [candidate] (50.019 ms) : 0, 50019
Remote Config [baseline] (677.222 µs) : 0, 677
Remote Config [candidate] (681.131 µs) : 0, 681
Telemetry [baseline] (7.673 ms) : 0, 7673
Telemetry [candidate] (7.556 ms) : 0, 7556
section iast
BytebuddyAgent [baseline] (779.006 ms) : 0, 779006
BytebuddyAgent [candidate] (777.772 ms) : 0, 777772
GlobalTracer [baseline] (295.452 ms) : 0, 295452
GlobalTracer [candidate] (293.381 ms) : 0, 293381
AppSec [baseline] (47.251 ms) : 0, 47251
AppSec [candidate] (46.949 ms) : 0, 46949
IAST [baseline] (26.238 ms) : 0, 26238
IAST [candidate] (28.802 ms) : 0, 28802
Remote Config [baseline] (573.489 µs) : 0, 573
Remote Config [candidate] (635.264 µs) : 0, 635
Telemetry [baseline] (7.017 ms) : 0, 7017
Telemetry [candidate] (6.877 ms) : 0, 6877
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (778.029 ms) : 0, 778029
BytebuddyAgent [candidate] (782.07 ms) : 0, 782070
GlobalTracer [baseline] (294.94 ms) : 0, 294940
GlobalTracer [candidate] (294.245 ms) : 0, 294245
AppSec [baseline] (47.103 ms) : 0, 47103
AppSec [candidate] (47.417 ms) : 0, 47417
IAST [baseline] (27.78 ms) : 0, 27780
IAST [candidate] (28.662 ms) : 0, 28662
Remote Config [baseline] (575.236 µs) : 0, 575
Remote Config [candidate] (586.246 µs) : 0, 586
Telemetry [baseline] (7.641 ms) : 0, 7641
Telemetry [candidate] (6.902 ms) : 0, 6902
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (776.27 ms) : 0, 776270
BytebuddyAgent [candidate] (778.377 ms) : 0, 778377
GlobalTracer [baseline] (295.531 ms) : 0, 295531
GlobalTracer [candidate] (294.225 ms) : 0, 294225
AppSec [baseline] (46.926 ms) : 0, 46926
AppSec [candidate] (47.25 ms) : 0, 47250
IAST [baseline] (27.729 ms) : 0, 27729
IAST [candidate] (25.154 ms) : 0, 25154
Remote Config [baseline] (600.481 µs) : 0, 600
Remote Config [candidate] (602.149 µs) : 0, 602
Telemetry [baseline] (7.557 ms) : 0, 7557
Telemetry [candidate] (7.656 ms) : 0, 7656
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.36.0-SNAPSHOT~d2672ea954, baseline=1.38.0-SNAPSHOT~120b584fde
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.076 s) : 0, 1076338
Total [baseline] (10.354 s) : 0, 10354111
Agent [candidate] (1.059 s) : 0, 1058661
Total [candidate] (10.338 s) : 0, 10338120
section appsec
Agent [baseline] (1.185 s) : 0, 1185134
Total [baseline] (10.513 s) : 0, 10513127
Agent [candidate] (1.185 s) : 0, 1184801
Total [candidate] (10.449 s) : 0, 10449029
section iast
Agent [baseline] (1.183 s) : 0, 1183015
Total [baseline] (10.771 s) : 0, 10771424
Agent [candidate] (1.164 s) : 0, 1163860
Total [candidate] (10.668 s) : 0, 10668050
section profiling
Agent [baseline] (1.265 s) : 0, 1265364
Total [baseline] (10.616 s) : 0, 10615924
Agent [candidate] (1.257 s) : 0, 1257472
Total [candidate] (10.597 s) : 0, 10597331
gantt
title petclinic - break down per module: candidate=1.36.0-SNAPSHOT~d2672ea954, baseline=1.38.0-SNAPSHOT~120b584fde
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (674.633 ms) : 0, 674633
BytebuddyAgent [candidate] (663.027 ms) : 0, 663027
GlobalTracer [baseline] (307.796 ms) : 0, 307796
GlobalTracer [candidate] (303.022 ms) : 0, 303022
AppSec [baseline] (50.432 ms) : 0, 50432
AppSec [candidate] (49.975 ms) : 0, 49975
Remote Config [baseline] (677.243 µs) : 0, 677
Remote Config [candidate] (680.186 µs) : 0, 680
Telemetry [baseline] (7.653 ms) : 0, 7653
Telemetry [candidate] (7.55 ms) : 0, 7550
section appsec
BytebuddyAgent [baseline] (678.018 ms) : 0, 678018
BytebuddyAgent [candidate] (677.76 ms) : 0, 677760
GlobalTracer [baseline] (300.006 ms) : 0, 300006
GlobalTracer [candidate] (298.191 ms) : 0, 298191
AppSec [baseline] (154.25 ms) : 0, 154250
AppSec [candidate] (154.165 ms) : 0, 154165
IAST [baseline] (18.967 ms) : 0, 18967
IAST [candidate] (22.42 ms) : 0, 22420
Remote Config [baseline] (622.317 µs) : 0, 622
Remote Config [candidate] (644.05 µs) : 0, 644
Telemetry [baseline] (7.998 ms) : 0, 7998
Telemetry [candidate] (8.155 ms) : 0, 8155
section iast
BytebuddyAgent [baseline] (786.907 ms) : 0, 786907
BytebuddyAgent [candidate] (776.297 ms) : 0, 776297
GlobalTracer [baseline] (298.635 ms) : 0, 298635
GlobalTracer [candidate] (292.742 ms) : 0, 292742
AppSec [baseline] (47.53 ms) : 0, 47530
AppSec [candidate] (47.041 ms) : 0, 47041
IAST [baseline] (27.852 ms) : 0, 27852
IAST [candidate] (26.881 ms) : 0, 26881
Remote Config [baseline] (585.149 µs) : 0, 585
Remote Config [candidate] (632.891 µs) : 0, 633
Telemetry [baseline] (7.855 ms) : 0, 7855
Telemetry [candidate] (7.002 ms) : 0, 7002
section profiling
BytebuddyAgent [baseline] (661.989 ms) : 0, 661989
BytebuddyAgent [candidate] (660.253 ms) : 0, 660253
GlobalTracer [baseline] (389.38 ms) : 0, 389380
GlobalTracer [candidate] (385.689 ms) : 0, 385689
AppSec [baseline] (51.999 ms) : 0, 51999
AppSec [candidate] (51.13 ms) : 0, 51130
Remote Config [baseline] (644.061 µs) : 0, 644
Remote Config [candidate] (733.271 µs) : 0, 733
Telemetry [baseline] (7.355 ms) : 0, 7355
Telemetry [candidate] (7.363 ms) : 0, 7363
ProfilingAgent [baseline] (96.731 ms) : 0, 96731
ProfilingAgent [candidate] (95.412 ms) : 0, 95412
Profiling [baseline] (96.756 ms) : 0, 96756
Profiling [candidate] (95.437 ms) : 0, 95437
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.36.0-SNAPSHOT~d2672ea954, baseline=1.38.0-SNAPSHOT~120b584fde
dateFormat X
axisFormat %s
section baseline
no_agent (1.337 ms) : 1317, 1356
. : milestone, 1337,
appsec (1.704 ms) : 1680, 1728
. : milestone, 1704,
appsec_no_iast (1.723 ms) : 1696, 1750
. : milestone, 1723,
iast (1.459 ms) : 1437, 1481
. : milestone, 1459,
profiling (1.524 ms) : 1498, 1550
. : milestone, 1524,
tracing (1.464 ms) : 1439, 1488
. : milestone, 1464,
section candidate
no_agent (1.354 ms) : 1335, 1373
. : milestone, 1354,
appsec (1.707 ms) : 1684, 1731
. : milestone, 1707,
appsec_no_iast (1.73 ms) : 1705, 1755
. : milestone, 1730,
iast (1.481 ms) : 1458, 1504
. : milestone, 1481,
profiling (1.488 ms) : 1465, 1512
. : milestone, 1488,
tracing (1.475 ms) : 1452, 1499
. : milestone, 1475,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.36.0-SNAPSHOT~d2672ea954, baseline=1.38.0-SNAPSHOT~120b584fde
dateFormat X
axisFormat %s
section baseline
no_agent (361.037 µs) : 341, 381
. : milestone, 361,
iast (486.277 µs) : 465, 508
. : milestone, 486,
iast_FULL (544.051 µs) : 523, 565
. : milestone, 544,
iast_GLOBAL (499.179 µs) : 478, 520
. : milestone, 499,
iast_HARDCODED_SECRET_DISABLED (486.008 µs) : 465, 507
. : milestone, 486,
iast_INACTIVE (454.642 µs) : 433, 476
. : milestone, 455,
iast_TELEMETRY_OFF (468.642 µs) : 447, 490
. : milestone, 469,
tracing (439.044 µs) : 418, 460
. : milestone, 439,
section candidate
no_agent (367.987 µs) : 349, 387
. : milestone, 368,
iast (479.303 µs) : 458, 500
. : milestone, 479,
iast_FULL (549.133 µs) : 528, 570
. : milestone, 549,
iast_GLOBAL (511.217 µs) : 489, 533
. : milestone, 511,
iast_HARDCODED_SECRET_DISABLED (485.352 µs) : 464, 507
. : milestone, 485,
iast_INACTIVE (452.38 µs) : 431, 473
. : milestone, 452,
iast_TELEMETRY_OFF (469.618 µs) : 448, 491
. : milestone, 470,
tracing (441.126 µs) : 421, 462
. : milestone, 441,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.36.0-SNAPSHOT~d2672ea954, baseline=1.38.0-SNAPSHOT~120b584fde
dateFormat X
axisFormat %s
section baseline
no_agent (1.46 ms) : 1449, 1472
. : milestone, 1460,
appsec (2.225 ms) : 2190, 2260
. : milestone, 2225,
iast (1.961 ms) : 1920, 2003
. : milestone, 1961,
iast_GLOBAL (2.02 ms) : 1978, 2061
. : milestone, 2020,
profiling (1.862 ms) : 1828, 1897
. : milestone, 1862,
tracing (1.844 ms) : 1811, 1877
. : milestone, 1844,
section candidate
no_agent (1.463 ms) : 1451, 1474
. : milestone, 1463,
appsec (2.192 ms) : 2160, 2225
. : milestone, 2192,
iast (1.948 ms) : 1909, 1988
. : milestone, 1948,
iast_GLOBAL (1.997 ms) : 1957, 2037
. : milestone, 1997,
profiling (1.846 ms) : 1814, 1878
. : milestone, 1846,
tracing (1.809 ms) : 1779, 1839
. : milestone, 1809,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.36.0-SNAPSHOT~d2672ea954, baseline=1.38.0-SNAPSHOT~120b584fde
dateFormat X
axisFormat %s
section baseline
no_agent (15.012 s) : 15012000, 15012000
. : milestone, 15012000,
appsec (14.807 s) : 14807000, 14807000
. : milestone, 14807000,
iast (19.062 s) : 19062000, 19062000
. : milestone, 19062000,
iast_GLOBAL (17.856 s) : 17856000, 17856000
. : milestone, 17856000,
profiling (15.044 s) : 15044000, 15044000
. : milestone, 15044000,
tracing (14.748 s) : 14748000, 14748000
. : milestone, 14748000,
section candidate
no_agent (15.364 s) : 15364000, 15364000
. : milestone, 15364000,
appsec (15.264 s) : 15264000, 15264000
. : milestone, 15264000,
iast (18.539 s) : 18539000, 18539000
. : milestone, 18539000,
iast_GLOBAL (17.785 s) : 17785000, 17785000
. : milestone, 17785000,
profiling (15.353 s) : 15353000, 15353000
. : milestone, 15353000,
tracing (14.894 s) : 14894000, 14894000
. : milestone, 14894000,
|
...er-openjdk/src/main/java/com/datadog/profiling/controller/openjdk/events/SmapEntryEvent.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please make the pattern a static final, otherwise LGTM
Note: the text format being parsed for extracting NMT annotations is expected to change in upcoming JDK releases |
e970de7
to
b03e8a2
Compare
What Does This Do
This is a new PR following an extension of #7107's original changes + my botching of the commit history.
This PR is responsible for reading from
procfs
'smaps
entries for JVMs running on Linux, and optionally using existing NMT region annotations in JDK22+ (for now) to identify where the RSS is coming from.Motivation
NMT within the JVM does not account for tracking of all memory used by the process itself, which means that the NMT timeseries metrics we report are underestimating the real memory usage of the JVM (i.e. from the JVM itself, or native libraries used via JNI). See the discrepancy in reported memory vs actual memory usage below; this is particularly problematic when running within containers, where the heap may be sized correctly relative to the available memory, but the overall usage exceeds what is available leading to an OOMKill.
Additional Notes
System.map
output and the MBean interactions are a little hacky, but prevent us from needing to dive into VMStructs or working on OpenJDK to start collecting this data. Open to feedback for cleaning this up.Jira ticket: PROF-9851