[WIP] Hive/Iceberg table metadata collection on Spark #8152
+443
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…l plan from spark 3
… in sql update and delete
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
| @@ -102,6 +104,8 @@ public abstract class AbstractDatadogSparkListener extends SparkListener { | |||
| protected final HashMap<Long, SparkPlanInfo> sqlPlans = new HashMap<>(); | |||
| private final HashMap<String, SparkListenerExecutorAdded> liveExecutors = new HashMap<>(); | |||
|
|
|||
| private final HashMap<Long, List<SparkSQLUtils.LineageDataset>> lineageDatasets = new HashMap<>(); | |||
There was a problem hiding this comment.
⚪ Code Quality Violation
Suggested change
| private final HashMap<Long, List<SparkSQLUtils.LineageDataset>> lineageDatasets = new HashMap<>(); | |
| private final Map<Long, List<SparkSQLUtils.LineageDataset>> lineageDatasets = new HashMap<>(); |
Avoid using a specific implementation type; use the more general Map instead (...read more)
Relying on particular implementation types, such as, HashSet or LinkedList can limit your adaptability to embrace alternative implementations in the future, particularly as your requirements change and your code needs to undergo changes.
It is recommended to opt for general types such as Set or List when declaring variables and parameters.
BenchmarksStartupParameters
See matching parameters
SummaryFound 7 performance improvements and 12 performance regressions! Performance is the same for 37 metrics, 7 unstable metrics.
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.44.0-SNAPSHOT~5b921e4444, baseline=1.45.0-SNAPSHOT~0c2f7ff792
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.068 s) : 0, 1068138
Total [baseline] (10.525 s) : 0, 10524925
Agent [candidate] (1.1 s) : 0, 1099903
Total [candidate] (10.5 s) : 0, 10500277
section appsec
Agent [baseline] (1.193 s) : 0, 1193106
Total [baseline] (10.712 s) : 0, 10711744
Agent [candidate] (1.234 s) : 0, 1233716
Total [candidate] (10.785 s) : 0, 10785403
section iast
Agent [baseline] (1.182 s) : 0, 1181539
Total [baseline] (10.947 s) : 0, 10946705
Agent [candidate] (1.23 s) : 0, 1229962
Total [candidate] (10.936 s) : 0, 10936102
section profiling
Agent [baseline] (1.273 s) : 0, 1272776
Total [baseline] (10.855 s) : 0, 10854868
Agent [candidate] (1.327 s) : 0, 1327285
Total [candidate] (10.858 s) : 0, 10858168
gantt
title petclinic - break down per module: candidate=1.44.0-SNAPSHOT~5b921e4444, baseline=1.45.0-SNAPSHOT~0c2f7ff792
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (722.508 ms) : 0, 722508
BytebuddyAgent [candidate] (700.153 ms) : 0, 700153
GlobalTracer [baseline] (258.817 ms) : 0, 258817
GlobalTracer [candidate] (320.651 ms) : 0, 320651
AppSec [baseline] (57.522 ms) : 0, 57522
AppSec [candidate] (54.769 ms) : 0, 54769
Remote Config [baseline] (733.947 µs) : 0, 734
Remote Config [candidate] (688.351 µs) : 0, 688
Telemetry [baseline] (13.418 ms) : 0, 13418
Telemetry [candidate] (9.88 ms) : 0, 9880
section appsec
BytebuddyAgent [baseline] (733.694 ms) : 0, 733694
BytebuddyAgent [candidate] (716.762 ms) : 0, 716762
GlobalTracer [baseline] (254.381 ms) : 0, 254381
GlobalTracer [candidate] (316.684 ms) : 0, 316684
AppSec [baseline] (171.746 ms) : 0, 171746
AppSec [candidate] (167.886 ms) : 0, 167886
Remote Config [baseline] (654.851 µs) : 0, 655
Remote Config [candidate] (681.402 µs) : 0, 681
Telemetry [baseline] (7.988 ms) : 0, 7988
Telemetry [candidate] (7.789 ms) : 0, 7789
IAST [baseline] (19.202 ms) : 0, 19202
IAST [candidate] (20.155 ms) : 0, 20155
section iast
BytebuddyAgent [baseline] (831.346 ms) : 0, 831346
BytebuddyAgent [candidate] (820.089 ms) : 0, 820089
GlobalTracer [baseline] (246.527 ms) : 0, 246527
GlobalTracer [candidate] (308.313 ms) : 0, 308313
AppSec [baseline] (58.301 ms) : 0, 58301
AppSec [candidate] (57.338 ms) : 0, 57338
Remote Config [baseline] (649.795 µs) : 0, 650
Remote Config [candidate] (638.774 µs) : 0, 639
Telemetry [baseline] (8.557 ms) : 0, 8557
Telemetry [candidate] (7.612 ms) : 0, 7612
IAST [baseline] (21.141 ms) : 0, 21141
IAST [candidate] (22.106 ms) : 0, 22106
section profiling
BytebuddyAgent [baseline] (702.141 ms) : 0, 702141
BytebuddyAgent [candidate] (694.432 ms) : 0, 694432
GlobalTracer [baseline] (371.319 ms) : 0, 371319
GlobalTracer [candidate] (436.747 ms) : 0, 436747
AppSec [baseline] (53.66 ms) : 0, 53660
AppSec [candidate] (53.9 ms) : 0, 53900
Remote Config [baseline] (652.107 µs) : 0, 652
Remote Config [candidate] (657.115 µs) : 0, 657
Telemetry [baseline] (7.782 ms) : 0, 7782
Telemetry [candidate] (7.738 ms) : 0, 7738
ProfilingAgent [baseline] (95.501 ms) : 0, 95501
ProfilingAgent [candidate] (94.385 ms) : 0, 94385
Profiling [baseline] (95.524 ms) : 0, 95524
Profiling [candidate] (94.409 ms) : 0, 94409
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.44.0-SNAPSHOT~5b921e4444, baseline=1.45.0-SNAPSHOT~0c2f7ff792
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1059172
Total [baseline] (8.651 s) : 0, 8651387
Agent [candidate] (1.101 s) : 0, 1100538
Total [candidate] (8.686 s) : 0, 8686303
section iast
Agent [baseline] (1.189 s) : 0, 1189123
Total [baseline] (9.252 s) : 0, 9252289
Agent [candidate] (1.22 s) : 0, 1219790
Total [candidate] (9.221 s) : 0, 9220565
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.19 s) : 0, 1189894
Total [baseline] (9.23 s) : 0, 9230080
Agent [candidate] (1.222 s) : 0, 1221567
Total [candidate] (9.17 s) : 0, 9169533
section iast_TELEMETRY_OFF
Agent [baseline] (1.195 s) : 0, 1194875
Total [baseline] (9.233 s) : 0, 9232999
Agent [candidate] (1.219 s) : 0, 1219357
Total [candidate] (9.206 s) : 0, 9205634
gantt
title insecure-bank - break down per module: candidate=1.44.0-SNAPSHOT~5b921e4444, baseline=1.45.0-SNAPSHOT~0c2f7ff792
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (714.91 ms) : 0, 714910
BytebuddyAgent [candidate] (701.405 ms) : 0, 701405
GlobalTracer [baseline] (257.388 ms) : 0, 257388
GlobalTracer [candidate] (319.707 ms) : 0, 319707
AppSec [baseline] (55.184 ms) : 0, 55184
AppSec [candidate] (54.869 ms) : 0, 54869
Remote Config [baseline] (723.803 µs) : 0, 724
Remote Config [candidate] (691.668 µs) : 0, 692
Telemetry [baseline] (15.819 ms) : 0, 15819
Telemetry [candidate] (10.001 ms) : 0, 10001
section iast
BytebuddyAgent [baseline] (836.731 ms) : 0, 836731
BytebuddyAgent [candidate] (812.529 ms) : 0, 812529
GlobalTracer [baseline] (248.132 ms) : 0, 248132
GlobalTracer [candidate] (306.774 ms) : 0, 306774
AppSec [baseline] (58.382 ms) : 0, 58382
AppSec [candidate] (57.621 ms) : 0, 57621
Remote Config [baseline] (690.478 µs) : 0, 690
Remote Config [candidate] (622.965 µs) : 0, 623
Telemetry [baseline] (8.704 ms) : 0, 8704
Telemetry [candidate] (7.544 ms) : 0, 7544
IAST [baseline] (21.441 ms) : 0, 21441
IAST [candidate] (20.933 ms) : 0, 20933
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (837.505 ms) : 0, 837505
BytebuddyAgent [candidate] (813.844 ms) : 0, 813844
GlobalTracer [baseline] (248.103 ms) : 0, 248103
GlobalTracer [candidate] (306.842 ms) : 0, 306842
AppSec [baseline] (58.545 ms) : 0, 58545
AppSec [candidate] (57.113 ms) : 0, 57113
Remote Config [baseline] (670.737 µs) : 0, 671
Remote Config [candidate] (614.079 µs) : 0, 614
Telemetry [baseline] (8.562 ms) : 0, 8562
Telemetry [candidate] (7.5 ms) : 0, 7500
IAST [baseline] (21.415 ms) : 0, 21415
IAST [candidate] (21.852 ms) : 0, 21852
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (841.784 ms) : 0, 841784
BytebuddyAgent [candidate] (811.883 ms) : 0, 811883
GlobalTracer [baseline] (249.51 ms) : 0, 249510
GlobalTracer [candidate] (306.753 ms) : 0, 306753
AppSec [baseline] (58.453 ms) : 0, 58453
AppSec [candidate] (57.34 ms) : 0, 57340
Remote Config [baseline] (642.364 µs) : 0, 642
Remote Config [candidate] (637.746 µs) : 0, 638
Telemetry [baseline] (8.448 ms) : 0, 8448
Telemetry [candidate] (7.455 ms) : 0, 7455
IAST [baseline] (20.762 ms) : 0, 20762
IAST [candidate] (21.479 ms) : 0, 21479
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 15 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.44.0-SNAPSHOT~5b921e4444, baseline=1.45.0-SNAPSHOT~0c2f7ff792
dateFormat X
axisFormat %s
section baseline
no_agent (1.366 ms) : 1346, 1385
. : milestone, 1366,
appsec (1.754 ms) : 1730, 1778
. : milestone, 1754,
appsec_no_iast (1.778 ms) : 1754, 1802
. : milestone, 1778,
iast (1.509 ms) : 1484, 1533
. : milestone, 1509,
profiling (1.536 ms) : 1511, 1560
. : milestone, 1536,
tracing (1.497 ms) : 1473, 1521
. : milestone, 1497,
section candidate
no_agent (1.368 ms) : 1348, 1387
. : milestone, 1368,
appsec (1.765 ms) : 1741, 1790
. : milestone, 1765,
appsec_no_iast (1.749 ms) : 1724, 1774
. : milestone, 1749,
iast (1.509 ms) : 1486, 1532
. : milestone, 1509,
profiling (1.55 ms) : 1526, 1574
. : milestone, 1550,
tracing (1.492 ms) : 1467, 1517
. : milestone, 1492,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.44.0-SNAPSHOT~5b921e4444, baseline=1.45.0-SNAPSHOT~0c2f7ff792
dateFormat X
axisFormat %s
section baseline
no_agent (377.875 µs) : 358, 398
. : milestone, 378,
iast (492.987 µs) : 472, 514
. : milestone, 493,
iast_FULL (658.873 µs) : 637, 681
. : milestone, 659,
iast_GLOBAL (527.983 µs) : 506, 550
. : milestone, 528,
iast_HARDCODED_SECRET_DISABLED (502.266 µs) : 480, 524
. : milestone, 502,
iast_INACTIVE (455.058 µs) : 434, 476
. : milestone, 455,
iast_TELEMETRY_OFF (487.63 µs) : 466, 509
. : milestone, 488,
tracing (453.077 µs) : 433, 474
. : milestone, 453,
section candidate
no_agent (378.494 µs) : 358, 399
. : milestone, 378,
iast (490.518 µs) : 469, 512
. : milestone, 491,
iast_FULL (655.255 µs) : 634, 677
. : milestone, 655,
iast_GLOBAL (526.801 µs) : 505, 548
. : milestone, 527,
iast_HARDCODED_SECRET_DISABLED (493.979 µs) : 473, 515
. : milestone, 494,
iast_INACTIVE (456.555 µs) : 436, 477
. : milestone, 457,
iast_TELEMETRY_OFF (490.504 µs) : 469, 512
. : milestone, 491,
tracing (456.262 µs) : 435, 478
. : milestone, 456,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.44.0-SNAPSHOT~5b921e4444, baseline=1.45.0-SNAPSHOT~0c2f7ff792
dateFormat X
axisFormat %s
section baseline
no_agent (1.462 ms) : 1451, 1473
. : milestone, 1462,
appsec (2.348 ms) : 2306, 2391
. : milestone, 2348,
iast (2.091 ms) : 2037, 2145
. : milestone, 2091,
iast_GLOBAL (2.145 ms) : 2091, 2199
. : milestone, 2145,
profiling (1.978 ms) : 1934, 2023
. : milestone, 1978,
tracing (1.927 ms) : 1886, 1969
. : milestone, 1927,
section candidate
no_agent (1.466 ms) : 1455, 1478
. : milestone, 1466,
appsec (2.334 ms) : 2292, 2375
. : milestone, 2334,
iast (2.078 ms) : 2025, 2130
. : milestone, 2078,
iast_GLOBAL (2.129 ms) : 2076, 2182
. : milestone, 2129,
profiling (1.945 ms) : 1902, 1987
. : milestone, 1945,
tracing (1.929 ms) : 1889, 1969
. : milestone, 1929,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.44.0-SNAPSHOT~5b921e4444, baseline=1.45.0-SNAPSHOT~0c2f7ff792
dateFormat X
axisFormat %s
section baseline
no_agent (14.97 s) : 14970000, 14970000
. : milestone, 14970000,
appsec (14.877 s) : 14877000, 14877000
. : milestone, 14877000,
iast (18.706 s) : 18706000, 18706000
. : milestone, 18706000,
iast_GLOBAL (18.048 s) : 18048000, 18048000
. : milestone, 18048000,
profiling (15.216 s) : 15216000, 15216000
. : milestone, 15216000,
tracing (15.087 s) : 15087000, 15087000
. : milestone, 15087000,
section candidate
no_agent (14.842 s) : 14842000, 14842000
. : milestone, 14842000,
appsec (14.86 s) : 14860000, 14860000
. : milestone, 14860000,
iast (18.97 s) : 18970000, 18970000
. : milestone, 18970000,
iast_GLOBAL (17.717 s) : 17717000, 17717000
. : milestone, 17717000,
profiling (15.127 s) : 15127000, 15127000
. : milestone, 15127000,
tracing (14.786 s) : 14786000, 14786000
. : milestone, 14786000,
|
…easier parsing downstream
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]