ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE) #3033

eliottness · 2024-12-12T16:20:40Z

What does this PR do?

This PR adds a new tracingAsTransport running mode for the tracer. This mode was created to disable APM biling but still allow customers to use other products that use APM as it's transport layer.

In short, what was done is:

Add the (*config).tracingAsTransport boolean field
Move the trace rate limit configuration value to the config struct for ease of use
Allow other parts of dd-trace-go than the tracer to set propagating tags (_dd.p.*)
When the env var DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED, do these:
- Add a global tag named _dd.apm.enabled=0
- disable trace metrics and runtime metrics
- disable agent side metrics with the Datadog-Client-Computed-Stats http header
- Set rate limit to 1 trace par minute as kind of heartbeat to make sure the service still appears in the UI
- Disable tracing propagation injection and extraction when the tag _dd.p.appsec=1 is not received upstream or set locally

All of this should make sure APM UI is disabled, but setting ManualKeep on certain traces will allow to bypass the rate limiter and will allow other products to still work when tracing-as-transport mode is enabled.

Motivation

This PR is the result of 1 approved RFC on ASM side and one draft RFC on APM side.

System-tests Run

https://github.com/DataDog/dd-trace-go/actions/runs/12397569796/job/34608280725

Reviewer's Checklist

Changed code has unit tests for its functionality at or near 100% coverage.
System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
There is a benchmark for any new code, or changes to existing code.
If this interacts with the agent in a new way, a system test has been added.
Add an appropriate team label so this PR gets put in the right place for the release notes.
Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.
For internal contributors, a matching PR should be created to the v2-dev branch and reviewed by @DataDog/apm-go.

Unsure? Have a question? Request a review!

pr-commenter · 2024-12-12T17:04:24Z

Benchmarks

Benchmark execution time: 2024-12-19 11:16:26

Comparing candidate commit 1a458ca in PR branch eliottness/APPSEC-56100/standalone-asm with baseline commit d15e61a in branch main.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 0 unstable metrics.

scenario:BenchmarkSetTagMetric-24

🟩 execution_time [-8.430ns; -5.970ns] or [-6.727%; -4.764%]

datadog-datadog-prod-us1 · 2024-12-13T14:45:45Z

Datadog Report

Branch report: eliottness/APPSEC-56100/standalone-asm
Commit report: 9f624a5
Test service: dd-trace-go

✅ 0 Failed, 2981 Passed, 24 Skipped, 2m 27.99s Total Time

ddtrace/tracer/option.go

Signed-off-by: Eliott Bouhana <[email protected]>

darccio

LGTM

darccio · 2024-12-18T19:50:20Z

internal/appsec/emitter/waf/run.go

@@ -53,7 +53,7 @@ func (op *ContextOperation) Run(eventReceiver dyngo.Operation, addrs waf.RunAddr
 	actions.SendActionEvents(eventReceiver, result.Actions)

 	if result.HasEvents() {
-		log.Debug("appsec: WAF detected a suspicious event")
+		dyngo.EmitData(op, &SecurityEvent{})


so meta indeed

Julio-Guerra · 2024-12-19T09:07:41Z

ddtrace/tracer/log.go

@@ -59,6 +59,7 @@ type startupInfo struct {
 	FeatureFlags                []string                     `json:"feature_flags"`
 	PropagationStyleInject      string                       `json:"propagation_style_inject"`  // Propagation style for inject
 	PropagationStyleExtract     string                       `json:"propagation_style_extract"` // Propagation style for extract
+	TracingAsTransport          bool                         `json:"tracing_as_transport"`      // Whether the tracer is disabled and other products are using it as a transport


Julio-Guerra · 2024-12-19T09:11:30Z

ddtrace/tracer/tracer.go

@@ -423,7 +426,9 @@ func (t *tracer) worker(tick <-chan time.Time) {
 			t.statsd.Incr("datadog.tracer.flush_triggered", []string{"reason:invoked"}, 1)
 			t.traceWriter.flush()
 			t.statsd.Flush()
-			t.stats.flushAndSend(time.Now(), withCurrentBucket)
+			if !t.config.tracingAsTransport {
+				t.stats.flushAndSend(time.Now(), withCurrentBucket)


what happens with the data in withCurrentBucket?
should we mute the emitter side of things too (this place being the receiver)?

the emitter side of things is muted via the change done to canComputeStats()

nit for the next PR to come but I suggest then to update t.stats methods to noop them when this is disabled

ddtrace/tracer/tracer.go

internal/appsec/listener/waf/waf.go

internal/appsec/listener/waf/tags.go

Julio-Guerra · 2024-12-19T09:22:17Z

internal/appsec/emitter/waf/run.go

@@ -53,7 +53,7 @@ func (op *ContextOperation) Run(eventReceiver dyngo.Operation, addrs waf.RunAddr
 	actions.SendActionEvents(eventReceiver, result.Actions)

 	if result.HasEvents() {
-		log.Debug("appsec: WAF detected a suspicious event")
+		dyngo.EmitData(op, &SecurityEvent{})


so meta indeed

internal/appsec/listener/waf/waf.go

Signed-off-by: Eliott Bouhana <[email protected]>

RomainMuller · 2024-12-19T10:03:05Z

contrib/net/http/roundtripper.go

@@ -14,7 +14,6 @@ import (
 	"strings"

 	"gopkg.in/DataDog/dd-trace-go.v1/appsec/events"
-


ddtrace/tracer/option.go

ddtrace/tracer/rules_sampler.go

ddtrace/tracer/option.go

internal/appsec/listener/waf/tags.go

Signed-off-by: Eliott Bouhana <[email protected]>

hannahkm · 2024-12-20T16:25:10Z

@eliottness Could you backport this PR into the v2 branch? Thanks! 🙏🏼

…3033) Signed-off-by: Eliott Bouhana <[email protected]>

Applied comments appsec: stop storing span tags, directly call span.SetTag (#3044) Signed-off-by: Eliott Bouhana <[email protected]> ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE) (#3033) Signed-off-by: Eliott Bouhana <[email protected]> fix: improving test logic for TestStreamSendsErrorCode to avoid flakiness (#3049) vuln: upgrade golang.org/x/{crypto,net} to non-vulnerable versions (#3050) contrib/miekg/dns: resolve flaky test in TestExchange* (#3045) ddtrace/tracer: report datadog.tracer.api.errors health metric (#3024) build(deps): bump google.golang.org/grpc from 1.64.0 to 1.64.1 (#3001) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rodrigo Argüello <[email protected]> ddtrace/tracer: Report datadog.tracer.queue.enqueued.traces as health metric (#3019) ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE) (#3033) Signed-off-by: Eliott Bouhana <[email protected]> fix: improving test logic for TestStreamSendsErrorCode to avoid flakiness (#3049) vuln: upgrade golang.org/x/{crypto,net} to non-vulnerable versions (#3050) contrib/miekg/dns: resolve flaky test in TestExchange* (#3045) ddtrace/tracer: report datadog.tracer.api.errors health metric (#3024) build(deps): bump google.golang.org/grpc from 1.64.0 to 1.64.1 (#3001) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Rodrigo Argüello <[email protected]> ddtrace/tracer: Report datadog.tracer.queue.enqueued.traces as health metric (#3019)

eliottness added the appsec label Dec 12, 2024

eliottness force-pushed the eliottness/APPSEC-56100/standalone-asm branch from dbd4501 to 8abea1d Compare December 12, 2024 16:20

eliottness force-pushed the eliottness/APPSEC-56100/standalone-asm branch from 8abea1d to 666c79a Compare December 13, 2024 14:45

eliottness mentioned this pull request Dec 17, 2024

appsec: stop storing span tags, directly call span.SetTag #3044

Merged

7 tasks

darccio reviewed Dec 18, 2024

View reviewed changes

ddtrace/tracer/option.go Outdated Show resolved Hide resolved

eliottness added 3 commits December 18, 2024 13:51

ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE)

417acf0

Signed-off-by: Eliott Bouhana <[email protected]>

add propagation parameters

ab2ab40

Signed-off-by: Eliott Bouhana <[email protected]>

fix misc issues

16c4012

Signed-off-by: Eliott Bouhana <[email protected]>

eliottness force-pushed the eliottness/APPSEC-56100/standalone-asm branch from ffc34c3 to 16c4012 Compare December 18, 2024 16:47

eliottness added 2 commits December 18, 2024 17:50

only do one run of APPSEC_STANDALONE in the system-tests CI

baa4784

Signed-off-by: Eliott Bouhana <[email protected]>

don't set the _sampling_priority tag manually

d400974

Signed-off-by: Eliott Bouhana <[email protected]>

eliottness marked this pull request as ready for review December 18, 2024 17:10

eliottness requested review from a team as code owners December 18, 2024 17:10

github-actions bot added the apm:ecosystem contrib/* related feature requests or bugs label Dec 18, 2024

darccio approved these changes Dec 18, 2024

View reviewed changes

Julio-Guerra reviewed Dec 19, 2024

View reviewed changes

Julio-Guerra requested changes Dec 19, 2024

View reviewed changes

RomainMuller mentioned this pull request Dec 19, 2024

[golang] Enable APPSEC_STANDALONE DataDog/system-tests#3679

Merged

10 tasks

suggestions from @Julio-Guerra

2be912c

Signed-off-by: Eliott Bouhana <[email protected]>

eliottness requested a review from Julio-Guerra December 19, 2024 10:09

RomainMuller reviewed Dec 19, 2024

View reviewed changes

eliottness added 2 commits December 19, 2024 11:27

rename rateLimit to rateLimitPerSecond

4bb0581

Signed-off-by: Eliott Bouhana <[email protected]>

suggestions from @RomainMuller

1a458ca

Signed-off-by: Eliott Bouhana <[email protected]>

eliottness requested a review from RomainMuller December 19, 2024 10:36

RomainMuller approved these changes Dec 19, 2024

View reviewed changes

Julio-Guerra approved these changes Dec 19, 2024

View reviewed changes

add test for _dd.p.appsec=1

a691f52

Signed-off-by: Eliott Bouhana <[email protected]>

eliottness enabled auto-merge (squash) December 19, 2024 12:50

eliottness merged commit 2dfcb2a into main Dec 19, 2024
179 of 181 checks passed

eliottness deleted the eliottness/APPSEC-56100/standalone-asm branch December 19, 2024 12:57

e-n-0 pushed a commit that referenced this pull request Dec 23, 2024

ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE) (#…

0b95329

…3033) Signed-off-by: Eliott Bouhana <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE) #3033

ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE) #3033

eliottness commented Dec 12, 2024 •

edited

Loading

pr-commenter bot commented Dec 12, 2024 •

edited

Loading

datadog-datadog-prod-us1 bot commented Dec 13, 2024 •

edited

Loading

darccio left a comment

darccio Dec 18, 2024

Julio-Guerra Dec 19, 2024

Julio-Guerra Dec 19, 2024

Julio-Guerra Dec 19, 2024

eliottness Dec 19, 2024

Julio-Guerra Dec 19, 2024

Julio-Guerra Dec 19, 2024

RomainMuller Dec 19, 2024

hannahkm commented Dec 20, 2024

		@@ -14,7 +14,6 @@ import (
		"strings"

		"gopkg.in/DataDog/dd-trace-go.v1/appsec/events"

ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE) #3033

ddtrace/tracer: Tracing as transport-only mode (APPSEC_STANDALONE) #3033

Conversation

eliottness commented Dec 12, 2024 • edited Loading

What does this PR do?

Motivation

System-tests Run

Reviewer's Checklist

pr-commenter bot commented Dec 12, 2024 • edited Loading

Benchmarks

scenario:BenchmarkSetTagMetric-24

datadog-datadog-prod-us1 bot commented Dec 13, 2024 • edited Loading

Datadog Report

darccio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hannahkm commented Dec 20, 2024

eliottness commented Dec 12, 2024 •

edited

Loading

pr-commenter bot commented Dec 12, 2024 •

edited

Loading

datadog-datadog-prod-us1 bot commented Dec 13, 2024 •

edited

Loading