Add multiple context extraction support #6138

PerfectSlayer · 2023-11-02T17:49:56Z

What Does This Do

Extract multiple propagated context according the new W3C trace context initiative,
Add DD_TRACE_PROPAGATION_EXTRACT_FIRST configuration to limit to the first valid one,
Propagate W3C tracestate if matches the extracted context, even if not a Datadog one,
Add span links of terminated contexts (terminated but not matching).

Motivation

This will improve W3C tracecontext standard compatibility.
W3C trace state will be extracted from W3C extracted context even using Datadog headers.

Additional Notes

An addition PR is coming to documentation repository for the new configuration key and behavior change.

Jira tickets:

Few comments that might help reviewing:

Context interpreters are now aware of the propagation style --> Required to handle specific W3C behavior
Same for extracted context --> Add a new parameter to constructor. I did not want to create another overloaded constructor as the constructor was not call that often (it mostly impacts tests and I am fine with updating them rather than creating multiple constructors that would make the code harder to read)
The new extract first trace context logic will only impact the CompoundExtractor --> The RFC has a diagram to help understand the expected behavior. I tried to comment it the most possible for futur maintainability.
W3C tracestate is added to PTags --> It should now be supported whatever the main propagation style, not only W3C. So original from W3CPTags was removed in favor of the new generic field.
W3CPtagsCodec now supports PTags (not only W3CPTags) for tracestate injection. --> I also add the 32 member limit check, with and without dd member presence.
W3CPTagsCodec now use the generic error field for inconsistent trace id --> I removed logic override to rely on the parent field
B3 extractor test are fixed --> They failed to extract partial tag content when trace id is invalid.
Moved propagation tags factory to the parent ContextInterpreter and make sure PTags are always defined (empty at least) --> PTags factory was the same for all context interpreter using it. Moving it to parent allow to make empty initialization if not done by the child interpreter. It does not introduce performance overhead as PTags will always be creating when creating a span from an extracted context (even an empty instance). Creating it at ExtractedContext level ensure we can propagate tracestate in CompoundExtractor.

pr-commenter · 2023-11-02T18:31:27Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
commit	1.23.0-SNAPSHOT~0a613832c1	1.23.0-SNAPSHOT~c1f32c9e2d
config	baseline	candidate

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 cases.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.23.0-SNAPSHOT~c1f32c9e2d, baseline=1.23.0-SNAPSHOT~0a613832c1

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.041 s) : 0, 1041368
Total [baseline] (9.288 s) : 0, 9288291
Agent [candidate] (1.039 s) : 0, 1039074
Total [candidate] (9.332 s) : 0, 9331605
section appsec
Agent [baseline] (1.117 s) : 0, 1117296
Total [baseline] (9.43 s) : 0, 9429785
Agent [candidate] (1.129 s) : 0, 1129080
Total [candidate] (9.373 s) : 0, 9373253
section iast
Agent [baseline] (1.15 s) : 0, 1149991
Total [baseline] (9.489 s) : 0, 9489078
Agent [candidate] (1.158 s) : 0, 1157832
Total [candidate] (9.551 s) : 0, 9551005
section profiling
Agent [baseline] (1.229 s) : 0, 1228824
Total [baseline] (9.528 s) : 0, 9527951
Agent [candidate] (1.22 s) : 0, 1219509
Total [candidate] (9.475 s) : 0, 9474820

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.041 s	-
Agent	appsec	1.117 s	75.927 ms (7.3%)
Agent	iast	1.15 s	108.623 ms (10.4%)
Agent	profiling	1.229 s	187.456 ms (18.0%)
Total	tracing	9.288 s	-
Total	appsec	9.43 s	141.494 ms (1.5%)
Total	iast	9.489 s	200.787 ms (2.2%)
Total	profiling	9.528 s	239.66 ms (2.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.039 s	-
Agent	appsec	1.129 s	90.005 ms (8.7%)
Agent	iast	1.158 s	118.758 ms (11.4%)
Agent	profiling	1.22 s	180.435 ms (17.4%)
Total	tracing	9.332 s	-
Total	appsec	9.373 s	41.648 ms (0.4%)
Total	iast	9.551 s	219.4 ms (2.4%)
Total	profiling	9.475 s	143.214 ms (1.5%)

gantt
    title petclinic - break down per module: candidate=1.23.0-SNAPSHOT~c1f32c9e2d, baseline=1.23.0-SNAPSHOT~0a613832c1

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (650.02 ms) : 0, 650020
BytebuddyAgent [candidate] (647.411 ms) : 0, 647411
GlobalTracer [baseline] (295.356 ms) : 0, 295356
GlobalTracer [candidate] (295.697 ms) : 0, 295697
AppSec [baseline] (49.13 ms) : 0, 49130
AppSec [candidate] (49.362 ms) : 0, 49362
Remote Config [baseline] (703.575 µs) : 0, 704
Remote Config [candidate] (676.779 µs) : 0, 677
Telemetry [baseline] (11.492 ms) : 0, 11492
Telemetry [candidate] (11.479 ms) : 0, 11479
section appsec
BytebuddyAgent [baseline] (644.62 ms) : 0, 644620
BytebuddyAgent [candidate] (651.468 ms) : 0, 651468
GlobalTracer [baseline] (293.043 ms) : 0, 293043
GlobalTracer [candidate] (296.389 ms) : 0, 296389
AppSec [baseline] (137.795 ms) : 0, 137795
AppSec [candidate] (139.007 ms) : 0, 139007
Remote Config [baseline] (645.686 µs) : 0, 646
Remote Config [candidate] (645.236 µs) : 0, 645
Telemetry [baseline] (6.862 ms) : 0, 6862
Telemetry [candidate] (6.923 ms) : 0, 6923
section iast
BytebuddyAgent [baseline] (764.437 ms) : 0, 764437
BytebuddyAgent [candidate] (770.964 ms) : 0, 770964
GlobalTracer [baseline] (274.181 ms) : 0, 274181
GlobalTracer [candidate] (276.693 ms) : 0, 276693
AppSec [baseline] (46.839 ms) : 0, 46839
AppSec [candidate] (46.793 ms) : 0, 46793
Remote Config [baseline] (571.73 µs) : 0, 572
Remote Config [candidate] (573.577 µs) : 0, 574
Telemetry [baseline] (13.985 ms) : 0, 13985
Telemetry [candidate] (12.064 ms) : 0, 12064
IAST [baseline] (15.6 ms) : 0, 15600
IAST [candidate] (16.007 ms) : 0, 16007
section profiling
ProfilingAgent [baseline] (89.594 ms) : 0, 89594
ProfilingAgent [candidate] (88.043 ms) : 0, 88043
BytebuddyAgent [baseline] (661.6 ms) : 0, 661600
BytebuddyAgent [candidate] (656.686 ms) : 0, 656686
GlobalTracer [baseline] (361.064 ms) : 0, 361064
GlobalTracer [candidate] (359.393 ms) : 0, 359393
AppSec [baseline] (49.579 ms) : 0, 49579
AppSec [candidate] (48.902 ms) : 0, 48902
Remote Config [baseline] (652.579 µs) : 0, 653
Remote Config [candidate] (649.543 µs) : 0, 650
Telemetry [baseline] (11.436 ms) : 0, 11436
Telemetry [candidate] (11.534 ms) : 0, 11534
Profiling [baseline] (89.618 ms) : 0, 89618
Profiling [candidate] (88.066 ms) : 0, 88066

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.23.0-SNAPSHOT~c1f32c9e2d, baseline=1.23.0-SNAPSHOT~0a613832c1

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.033 s) : 0, 1032752
Total [baseline] (8.75 s) : 0, 8749974
Agent [candidate] (1.036 s) : 0, 1036351
Total [candidate] (8.773 s) : 0, 8772504
section iast
Agent [baseline] (1.156 s) : 0, 1155909
Total [baseline] (9.314 s) : 0, 9313672
Agent [candidate] (1.152 s) : 0, 1152294
Total [candidate] (9.324 s) : 0, 9323509
section iast_TELEMETRY_OFF
Agent [baseline] (1.153 s) : 0, 1153295
Total [baseline] (9.301 s) : 0, 9301196
Agent [candidate] (1.146 s) : 0, 1146480
Total [candidate] (9.275 s) : 0, 9275267

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.033 s	-
Agent	iast	1.156 s	123.157 ms (11.9%)
Agent	iast_TELEMETRY_OFF	1.153 s	120.543 ms (11.7%)
Total	tracing	8.75 s	-
Total	iast	9.314 s	563.698 ms (6.4%)
Total	iast_TELEMETRY_OFF	9.301 s	551.222 ms (6.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.036 s	-
Agent	iast	1.152 s	115.943 ms (11.2%)
Agent	iast_TELEMETRY_OFF	1.146 s	110.129 ms (10.6%)
Total	tracing	8.773 s	-
Total	iast	9.324 s	551.006 ms (6.3%)
Total	iast_TELEMETRY_OFF	9.275 s	502.763 ms (5.7%)

gantt
    title insecure-bank - break down per module: candidate=1.23.0-SNAPSHOT~c1f32c9e2d, baseline=1.23.0-SNAPSHOT~0a613832c1

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (644.626 ms) : 0, 644626
BytebuddyAgent [candidate] (645.598 ms) : 0, 645598
GlobalTracer [baseline] (293.011 ms) : 0, 293011
GlobalTracer [candidate] (295.175 ms) : 0, 295175
AppSec [baseline] (48.625 ms) : 0, 48625
AppSec [candidate] (49.064 ms) : 0, 49064
Remote Config [baseline] (697.306 µs) : 0, 697
Remote Config [candidate] (682.349 µs) : 0, 682
Telemetry [baseline] (11.296 ms) : 0, 11296
Telemetry [candidate] (11.359 ms) : 0, 11359
section iast
BytebuddyAgent [baseline] (770.223 ms) : 0, 770223
BytebuddyAgent [candidate] (765.875 ms) : 0, 765875
GlobalTracer [baseline] (274.27 ms) : 0, 274270
GlobalTracer [candidate] (275.844 ms) : 0, 275844
AppSec [baseline] (46.722 ms) : 0, 46722
AppSec [candidate] (46.707 ms) : 0, 46707
IAST [baseline] (16.106 ms) : 0, 16106
IAST [candidate] (16.655 ms) : 0, 16655
Remote Config [baseline] (560.573 µs) : 0, 561
Remote Config [candidate] (578.656 µs) : 0, 579
Telemetry [baseline] (13.307 ms) : 0, 13307
Telemetry [candidate] (12.116 ms) : 0, 12116
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (763.804 ms) : 0, 763804
BytebuddyAgent [candidate] (759.222 ms) : 0, 759222
GlobalTracer [baseline] (276.462 ms) : 0, 276462
GlobalTracer [candidate] (275.685 ms) : 0, 275685
AppSec [baseline] (47.04 ms) : 0, 47040
AppSec [candidate] (46.665 ms) : 0, 46665
IAST [baseline] (18.267 ms) : 0, 18267
IAST [candidate] (18.669 ms) : 0, 18669
Remote Config [baseline] (580.435 µs) : 0, 580
Remote Config [candidate] (599.835 µs) : 0, 600
Telemetry [baseline] (12.555 ms) : 0, 12555
Telemetry [candidate] (11.176 ms) : 0, 11176

Load

Parameters

	Baseline	Candidate
commit	1.23.0-SNAPSHOT~0a613832c1	1.23.0-SNAPSHOT~c1f32c9e2d
config	baseline	candidate
end_time	2023-11-08T08:55:15	2023-11-08T09:11:42
start_time	2023-11-08T08:55:02	2023-11-08T09:11:29

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 22 cases.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.23.0-SNAPSHOT~c1f32c9e2d, baseline=1.23.0-SNAPSHOT~0a613832c1
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.357 ms) : 1338, 1376
.   : milestone, 1357,
appsec (1.693 ms) : 1668, 1718
.   : milestone, 1693,
iast (1.491 ms) : 1467, 1515
.   : milestone, 1491,
profiling (1.469 ms) : 1444, 1494
.   : milestone, 1469,
tracing (1.443 ms) : 1418, 1468
.   : milestone, 1443,
section candidate
no_agent (1.338 ms) : 1319, 1357
.   : milestone, 1338,
appsec (1.724 ms) : 1699, 1748
.   : milestone, 1724,
iast (1.464 ms) : 1440, 1488
.   : milestone, 1464,
profiling (1.463 ms) : 1438, 1487
.   : milestone, 1463,
tracing (1.471 ms) : 1447, 1495
.   : milestone, 1471,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.357 ms [1.338 ms, 1.376 ms]	-
appsec	1.693 ms [1.668 ms, 1.718 ms]	336.358 µs (24.8%)
iast	1.491 ms [1.467 ms, 1.515 ms]	133.989 µs (9.9%)
profiling	1.469 ms [1.444 ms, 1.494 ms]	112.21 µs (8.3%)
tracing	1.443 ms [1.418 ms, 1.468 ms]	85.975 µs (6.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.338 ms [1.319 ms, 1.357 ms]	-
appsec	1.724 ms [1.699 ms, 1.748 ms]	386.167 µs (28.9%)
iast	1.464 ms [1.44 ms, 1.488 ms]	126.056 µs (9.4%)
profiling	1.463 ms [1.438 ms, 1.487 ms]	125.197 µs (9.4%)
tracing	1.471 ms [1.447 ms, 1.495 ms]	133.271 µs (10.0%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.23.0-SNAPSHOT~c1f32c9e2d, baseline=1.23.0-SNAPSHOT~0a613832c1
    dateFormat X
    axisFormat %s
section baseline
no_agent (360.98 µs) : 340, 382
.   : milestone, 361,
iast (456.957 µs) : 436, 478
.   : milestone, 457,
iast_FULL (525.054 µs) : 504, 546
.   : milestone, 525,
iast_INACTIVE (440.818 µs) : 420, 462
.   : milestone, 441,
iast_TELEMETRY_OFF (459.434 µs) : 438, 481
.   : milestone, 459,
tracing (432.332 µs) : 411, 454
.   : milestone, 432,
section candidate
no_agent (364.679 µs) : 344, 385
.   : milestone, 365,
iast (461.747 µs) : 441, 483
.   : milestone, 462,
iast_FULL (527.755 µs) : 507, 549
.   : milestone, 528,
iast_INACTIVE (438.185 µs) : 417, 459
.   : milestone, 438,
iast_TELEMETRY_OFF (462.084 µs) : 441, 484
.   : milestone, 462,
tracing (437.619 µs) : 416, 460
.   : milestone, 438,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	360.98 µs [340.407 µs, 381.552 µs]	-
iast	456.957 µs [436.403 µs, 477.512 µs]	95.978 µs (26.6%)
iast_FULL	525.054 µs [504.33 µs, 545.778 µs]	164.075 µs (45.5%)
iast_INACTIVE	440.818 µs [419.649 µs, 461.987 µs]	79.838 µs (22.1%)
iast_TELEMETRY_OFF	459.434 µs [438.297 µs, 480.571 µs]	98.454 µs (27.3%)
tracing	432.332 µs [410.902 µs, 453.762 µs]	71.352 µs (19.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	364.679 µs [344.082 µs, 385.276 µs]	-
iast	461.747 µs [440.839 µs, 482.655 µs]	97.068 µs (26.6%)
iast_FULL	527.755 µs [506.962 µs, 548.549 µs]	163.077 µs (44.7%)
iast_INACTIVE	438.185 µs [417.0 µs, 459.371 µs]	73.507 µs (20.2%)
iast_TELEMETRY_OFF	462.084 µs [440.623 µs, 483.546 µs]	97.405 µs (26.7%)
tracing	437.619 µs [415.662 µs, 459.577 µs]	72.94 µs (20.0%)

PerfectSlayer · 2023-11-06T17:22:06Z

I marked the PR ready for review as it should be feature compete and pass the related system tests (currently in development).
It still need to performance measurement though.

Extract multiple propagated context according the new W3C trace context initiative. Add a config to limit to the first valid one.

… present using W3CPtags

…ags instance It will prevent the only case where a DDSpanContext can have a null PTags. This will allow to store tracestate if found from another valid extracted context.

mcculls

LGTM

PerfectSlayer added type: enhancement comp: context propagation Trace context propagation labels Nov 2, 2023

PerfectSlayer changed the base branch from master to project/trace-context November 3, 2023 15:30

PerfectSlayer force-pushed the bbujon/trace-context-multiple-extractors branch 2 times, most recently from a015c11 to 1ae9781 Compare November 6, 2023 16:59

PerfectSlayer marked this pull request as ready for review November 6, 2023 17:20

PerfectSlayer requested a review from a team as a code owner November 6, 2023 17:20

PerfectSlayer requested review from mcculls, am312 and nayeem-kamal November 6, 2023 17:20

PerfectSlayer added 9 commits November 7, 2023 16:24

feat(core): Add multiple context extraction

a70d27b

Extract multiple propagated context according the new W3C trace context initiative. Add a config to limit to the first valid one.

fix(core): Fix B3 extractor tests

7ebe31d

feat(core): Add terminated extracted contexts as span links

df0a27f

feat(core): Add parent span to tracestate

c63ce70

feat(core): Revert parent span addition to tracestate

cbecc27

feat(core): Clean up W3C propagation tags error handling

e518431

feat(core): Add tracestate propagation from different propagation tags

c7bc842

fix(core): Fix tracestate encoding when Datadog and other members are…

ca90032

… present using W3CPtags

fix(core): Fix terminated context addition in span builder

f0fa179

PerfectSlayer force-pushed the bbujon/trace-context-multiple-extractors branch from 1ae9781 to f0fa179 Compare November 7, 2023 15:26

PerfectSlayer added 3 commits November 7, 2023 17:45

fix(core): Fix trace id comparison

5a3e9b7

chore: Clean up comments

2692631

feat(core): Ensure extracted context always have at least an empty PT…

c1f32c9

…ags instance It will prevent the only case where a DDSpanContext can have a null PTags. This will allow to store tracestate if found from another valid extracted context.

mcculls approved these changes Nov 8, 2023

View reviewed changes

PerfectSlayer merged commit 30eeaf5 into project/trace-context Nov 8, 2023
56 of 60 checks passed

PerfectSlayer deleted the bbujon/trace-context-multiple-extractors branch November 8, 2023 13:04

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add multiple context extraction support #6138

Add multiple context extraction support #6138

PerfectSlayer commented Nov 2, 2023 •

edited

Loading

pr-commenter bot commented Nov 2, 2023 •

edited

Loading

PerfectSlayer commented Nov 6, 2023

mcculls left a comment

Add multiple context extraction support #6138

Add multiple context extraction support #6138

Conversation

PerfectSlayer commented Nov 2, 2023 • edited Loading

What Does This Do

Motivation

Additional Notes

pr-commenter bot commented Nov 2, 2023 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

PerfectSlayer commented Nov 6, 2023

mcculls left a comment

Choose a reason for hiding this comment

PerfectSlayer commented Nov 2, 2023 •

edited

Loading

pr-commenter bot commented Nov 2, 2023 •

edited

Loading