[MLOB] add LLM obs configs #8076
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gary-huang
force-pushed
the
gary/add-llmobs-configs
branch
from
3863387 to
19d2dc3
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 4 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~14f71b3d0a, baseline=1.45.0-SNAPSHOT~a8b33d5c49
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1048661
Total [baseline] (8.629 s) : 0, 8629077
Agent [candidate] (1.055 s) : 0, 1054549
Total [candidate] (8.649 s) : 0, 8648590
section iast
Agent [baseline] (1.188 s) : 0, 1188126
Total [baseline] (9.228 s) : 0, 9227928
Agent [candidate] (1.185 s) : 0, 1185409
Total [candidate] (9.273 s) : 0, 9272948
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.18 s) : 0, 1180135
Total [baseline] (9.175 s) : 0, 9175018
Agent [candidate] (1.178 s) : 0, 1177939
Total [candidate] (9.191 s) : 0, 9191238
section iast_TELEMETRY_OFF
Agent [baseline] (1.177 s) : 0, 1176501
Total [baseline] (9.186 s) : 0, 9185889
Agent [candidate] (1.173 s) : 0, 1173245
Total [candidate] (9.22 s) : 0, 9220304
gantt
title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~14f71b3d0a, baseline=1.45.0-SNAPSHOT~a8b33d5c49
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (710.239 ms) : 0, 710239
BytebuddyAgent [candidate] (712.111 ms) : 0, 712111
GlobalTracer [baseline] (255.824 ms) : 0, 255824
GlobalTracer [candidate] (256.042 ms) : 0, 256042
AppSec [baseline] (54.959 ms) : 0, 54959
AppSec [candidate] (55.788 ms) : 0, 55788
Remote Config [baseline] (711.909 µs) : 0, 712
Remote Config [candidate] (695.864 µs) : 0, 696
Telemetry [baseline] (11.96 ms) : 0, 11960
Telemetry [candidate] (14.968 ms) : 0, 14968
section iast
BytebuddyAgent [baseline] (835.36 ms) : 0, 835360
BytebuddyAgent [candidate] (833.884 ms) : 0, 833884
GlobalTracer [baseline] (247.685 ms) : 0, 247685
GlobalTracer [candidate] (247.525 ms) : 0, 247525
AppSec [baseline] (58.799 ms) : 0, 58799
AppSec [candidate] (58.298 ms) : 0, 58298
IAST [baseline] (21.722 ms) : 0, 21722
IAST [candidate] (21.454 ms) : 0, 21454
Remote Config [baseline] (665.818 µs) : 0, 666
Remote Config [candidate] (675.678 µs) : 0, 676
Telemetry [baseline] (8.757 ms) : 0, 8757
Telemetry [candidate] (8.522 ms) : 0, 8522
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (829.628 ms) : 0, 829628
BytebuddyAgent [candidate] (827.879 ms) : 0, 827879
GlobalTracer [baseline] (247.063 ms) : 0, 247063
GlobalTracer [candidate] (247.129 ms) : 0, 247129
AppSec [baseline] (58.063 ms) : 0, 58063
AppSec [candidate] (57.969 ms) : 0, 57969
IAST [baseline] (21.155 ms) : 0, 21155
IAST [candidate] (20.976 ms) : 0, 20976
Remote Config [baseline] (647.753 µs) : 0, 648
Remote Config [candidate] (640.206 µs) : 0, 640
Telemetry [baseline] (8.535 ms) : 0, 8535
Telemetry [candidate] (8.447 ms) : 0, 8447
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (826.783 ms) : 0, 826783
BytebuddyAgent [candidate] (824.67 ms) : 0, 824670
GlobalTracer [baseline] (246.632 ms) : 0, 246632
GlobalTracer [candidate] (246.185 ms) : 0, 246185
AppSec [baseline] (57.772 ms) : 0, 57772
AppSec [candidate] (57.804 ms) : 0, 57804
IAST [baseline] (21.252 ms) : 0, 21252
IAST [candidate] (20.679 ms) : 0, 20679
Remote Config [baseline] (654.501 µs) : 0, 655
Remote Config [candidate] (628.988 µs) : 0, 629
Telemetry [baseline] (8.446 ms) : 0, 8446
Telemetry [candidate] (8.319 ms) : 0, 8319
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~14f71b3d0a, baseline=1.45.0-SNAPSHOT~a8b33d5c49
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.051 s) : 0, 1050746
Total [baseline] (10.429 s) : 0, 10428812
Agent [candidate] (1.058 s) : 0, 1058319
Total [candidate] (10.379 s) : 0, 10378511
section appsec
Agent [baseline] (1.193 s) : 0, 1193301
Total [baseline] (10.692 s) : 0, 10692076
Agent [candidate] (1.192 s) : 0, 1192475
Total [candidate] (10.707 s) : 0, 10707133
section iast
Agent [baseline] (1.179 s) : 0, 1179308
Total [baseline] (11.033 s) : 0, 11032844
Agent [candidate] (1.178 s) : 0, 1177569
Total [candidate] (11.024 s) : 0, 11024348
section profiling
Agent [baseline] (1.279 s) : 0, 1279290
Total [baseline] (10.822 s) : 0, 10822077
Agent [candidate] (1.287 s) : 0, 1286773
Total [candidate] (10.841 s) : 0, 10840954
gantt
title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~14f71b3d0a, baseline=1.45.0-SNAPSHOT~a8b33d5c49
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (712.604 ms) : 0, 712604
BytebuddyAgent [candidate] (716.901 ms) : 0, 716901
GlobalTracer [baseline] (256.559 ms) : 0, 256559
GlobalTracer [candidate] (256.367 ms) : 0, 256367
AppSec [baseline] (55.256 ms) : 0, 55256
AppSec [candidate] (58.303 ms) : 0, 58303
Remote Config [baseline] (721.776 µs) : 0, 722
Remote Config [candidate] (685.33 µs) : 0, 685
Telemetry [baseline] (10.629 ms) : 0, 10629
Telemetry [candidate] (11.069 ms) : 0, 11069
section appsec
BytebuddyAgent [baseline] (733.93 ms) : 0, 733930
BytebuddyAgent [candidate] (733.689 ms) : 0, 733689
GlobalTracer [baseline] (254.599 ms) : 0, 254599
GlobalTracer [candidate] (254.025 ms) : 0, 254025
AppSec [baseline] (170.609 ms) : 0, 170609
AppSec [candidate] (171.279 ms) : 0, 171279
IAST [baseline] (20.832 ms) : 0, 20832
IAST [candidate] (19.392 ms) : 0, 19392
Remote Config [baseline] (661.176 µs) : 0, 661
Remote Config [candidate] (668.505 µs) : 0, 669
Telemetry [baseline] (8.021 ms) : 0, 8021
Telemetry [candidate] (8.046 ms) : 0, 8046
section iast
BytebuddyAgent [baseline] (829.469 ms) : 0, 829469
BytebuddyAgent [candidate] (828.137 ms) : 0, 828137
GlobalTracer [baseline] (246.66 ms) : 0, 246660
GlobalTracer [candidate] (245.981 ms) : 0, 245981
AppSec [baseline] (57.814 ms) : 0, 57814
AppSec [candidate] (58.189 ms) : 0, 58189
IAST [baseline] (21.106 ms) : 0, 21106
IAST [candidate] (21.23 ms) : 0, 21230
Remote Config [baseline] (653.214 µs) : 0, 653
Remote Config [candidate] (665.579 µs) : 0, 666
Telemetry [baseline] (8.584 ms) : 0, 8584
Telemetry [candidate] (8.432 ms) : 0, 8432
section profiling
BytebuddyAgent [baseline] (704.356 ms) : 0, 704356
BytebuddyAgent [candidate] (711.129 ms) : 0, 711129
GlobalTracer [baseline] (373.466 ms) : 0, 373466
GlobalTracer [candidate] (374.412 ms) : 0, 374412
AppSec [baseline] (54.769 ms) : 0, 54769
AppSec [candidate] (54.057 ms) : 0, 54057
Remote Config [baseline] (667.1 µs) : 0, 667
Remote Config [candidate] (655.227 µs) : 0, 655
Telemetry [baseline] (7.832 ms) : 0, 7832
Telemetry [candidate] (7.799 ms) : 0, 7799
ProfilingAgent [baseline] (96.481 ms) : 0, 96481
ProfilingAgent [candidate] (96.367 ms) : 0, 96367
Profiling [baseline] (96.505 ms) : 0, 96505
Profiling [candidate] (96.392 ms) : 0, 96392
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 15 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~14f71b3d0a, baseline=1.45.0-SNAPSHOT~a8b33d5c49
dateFormat X
axisFormat %s
section baseline
no_agent (1.35 ms) : 1330, 1370
. : milestone, 1350,
appsec (1.753 ms) : 1730, 1776
. : milestone, 1753,
appsec_no_iast (1.767 ms) : 1744, 1790
. : milestone, 1767,
iast (1.507 ms) : 1484, 1530
. : milestone, 1507,
profiling (1.538 ms) : 1513, 1563
. : milestone, 1538,
tracing (1.494 ms) : 1469, 1519
. : milestone, 1494,
section candidate
no_agent (1.368 ms) : 1349, 1388
. : milestone, 1368,
appsec (1.755 ms) : 1731, 1779
. : milestone, 1755,
appsec_no_iast (1.76 ms) : 1735, 1785
. : milestone, 1760,
iast (1.515 ms) : 1491, 1539
. : milestone, 1515,
profiling (1.577 ms) : 1553, 1602
. : milestone, 1577,
tracing (1.508 ms) : 1483, 1533
. : milestone, 1508,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~14f71b3d0a, baseline=1.45.0-SNAPSHOT~a8b33d5c49
dateFormat X
axisFormat %s
section baseline
no_agent (383.589 µs) : 364, 403
. : milestone, 384,
iast (507.009 µs) : 485, 529
. : milestone, 507,
iast_FULL (662.982 µs) : 641, 685
. : milestone, 663,
iast_GLOBAL (535.646 µs) : 513, 559
. : milestone, 536,
iast_HARDCODED_SECRET_DISABLED (498.57 µs) : 477, 520
. : milestone, 499,
iast_INACTIVE (458.861 µs) : 438, 480
. : milestone, 459,
iast_TELEMETRY_OFF (496.44 µs) : 475, 518
. : milestone, 496,
tracing (470.52 µs) : 450, 491
. : milestone, 471,
section candidate
no_agent (373.43 µs) : 354, 393
. : milestone, 373,
iast (504.459 µs) : 483, 526
. : milestone, 504,
iast_FULL (665.148 µs) : 643, 687
. : milestone, 665,
iast_GLOBAL (524.711 µs) : 503, 547
. : milestone, 525,
iast_HARDCODED_SECRET_DISABLED (499.097 µs) : 477, 521
. : milestone, 499,
iast_INACTIVE (463.531 µs) : 442, 485
. : milestone, 464,
iast_TELEMETRY_OFF (490.245 µs) : 468, 512
. : milestone, 490,
tracing (460.425 µs) : 439, 482
. : milestone, 460,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~14f71b3d0a, baseline=1.45.0-SNAPSHOT~a8b33d5c49
dateFormat X
axisFormat %s
section baseline
no_agent (14.922 s) : 14922000, 14922000
. : milestone, 14922000,
appsec (15.195 s) : 15195000, 15195000
. : milestone, 15195000,
iast (18.929 s) : 18929000, 18929000
. : milestone, 18929000,
iast_GLOBAL (17.833 s) : 17833000, 17833000
. : milestone, 17833000,
profiling (15.547 s) : 15547000, 15547000
. : milestone, 15547000,
tracing (15.142 s) : 15142000, 15142000
. : milestone, 15142000,
section candidate
no_agent (15.586 s) : 15586000, 15586000
. : milestone, 15586000,
appsec (15.096 s) : 15096000, 15096000
. : milestone, 15096000,
iast (19.049 s) : 19049000, 19049000
. : milestone, 19049000,
iast_GLOBAL (17.676 s) : 17676000, 17676000
. : milestone, 17676000,
profiling (14.995 s) : 14995000, 14995000
. : milestone, 14995000,
tracing (14.91 s) : 14910000, 14910000
. : milestone, 14910000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~14f71b3d0a, baseline=1.45.0-SNAPSHOT~a8b33d5c49
dateFormat X
axisFormat %s
section baseline
no_agent (1.481 ms) : 1470, 1493
. : milestone, 1481,
appsec (2.367 ms) : 2325, 2409
. : milestone, 2367,
iast (2.109 ms) : 2056, 2162
. : milestone, 2109,
iast_GLOBAL (2.138 ms) : 2085, 2192
. : milestone, 2138,
profiling (1.967 ms) : 1925, 2009
. : milestone, 1967,
tracing (1.939 ms) : 1898, 1980
. : milestone, 1939,
section candidate
no_agent (1.474 ms) : 1463, 1486
. : milestone, 1474,
appsec (2.345 ms) : 2304, 2386
. : milestone, 2345,
iast (2.091 ms) : 2038, 2144
. : milestone, 2091,
iast_GLOBAL (2.152 ms) : 2099, 2206
. : milestone, 2152,
profiling (1.966 ms) : 1924, 2008
. : milestone, 1966,
tracing (1.938 ms) : 1897, 1979
. : milestone, 1938,
|
gary-huang
force-pushed
the
gary/add-llmobs-configs
branch
2 times, most recently
from
fd136e0 to
ba03bdd
Compare
gary-huang
force-pushed
the
gary/add-llmobs-configs
branch
from
ba03bdd to
a8108c6
Compare
|
Hi! 👋 Looks like you updated a Git Submodule.
|
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
Yun-Kim
approved these changes
Dec 30, 2024
Comment on lines
+2673
to
+2678
| public boolean isLlmObsAgentlessEnabled() { | ||
| return llmObsAgentlessEnabled; | ||
| } | ||
|
|
||
| public String getLlmObsMlApp() { | ||
| return llmObsMlApp; |
There was a problem hiding this comment.
Are these two options not set on the instrumenterConfig object? We're fine to just store this directly on the Config instance?
There was a problem hiding this comment.
i think it is fine to store these in the config instance for now, i am not sure but it is possible we may need them, instrumenterConfig seems to be about starting actual implementations classes if a product is enabled, i think we can add them later as needed
There was a problem hiding this comment.
Yup that's fine with me to add them later when needed. Thanks for clarifying
| */ | ||
| public final class LlmObsConfig { | ||
|
|
||
| public static final String LLM_OBS_ENABLED = "llmobs.enabled"; |
There was a problem hiding this comment.
Tiny nit, I think the other two language SDKs use the naming pattern LLMOBS_ instead of LLM_OBS_, would be nice to stay consistent but not that serious so up to you
There was a problem hiding this comment.
yes for sure, good catch!
|
Hi! 👋 Looks like you updated a Git Submodule.
|
|
Hi! 👋 Looks like you updated a Git Submodule.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
adds llm observability related configs and some start up behavior that are related to the enablement of llm observability
dd.llmobs.enabledis set to true and there is no ml app defined - throw errordd.llmobs.agentless.enabledis set to true and there is no API key supplied - throw errorjava system property success cases
agent mode configuration
agentless mode configuration with API key env var
env variable success cases
agent mode configuration
agentless mode configuration
Motivation
Additional Notes
tested with a sample spring app
the following stack satisfies 1
the follow stack satisfies 2
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]