[BUG] fatal error concurrent map iteration of ctx.baggage with tracer inject

[elijahj19]

Version of dd-trace-go

v1.59.0

Describe what happened:

Hi team, a project using this package 1.57.0 (later upgraded to 1.59.0) began to crash because of concurrent map iteration.

In all cases of the Fatal concurrent map errors, the stack traces of crashes showed tracer.Inject and mentioned this line here in which there is a raw map iteration of ctx.baggage (textmap.go#L367).

However, ctx.baggage should be protected by a mutex as commented here (spancontext.go#L99). Seemingly the concurrent safe ForeachBaggageItem should be used or the mutex should be locked instead of a raw iteration.

Describe what you expected:
No concurrent map iteration/write. Accessing map using mutex or concurrent safe methods.

Steps to reproduce the issue:

I've been able to locally reproduce the data race with the following code and running go run -race main.go

tracer.Start()
span, _ := tracer.StartSpanFromContext(context.Background(), "main")
defer span.Finish()

var wg sync.WaitGroup
for i := 0; i < 500; i++ {
  wg.Add(1)
  i := i
  go func(val int) {
    defer wg.Done()
    span.SetBaggageItem("val", fmt.Sprintf("%d", val))

    traceContext := map[string]string{}
    _ = tracer.Inject(span.Context(), tracer.TextMapCarrier(traceContext))
  }(i)
}

wg.Wait()

Additional environment details (Version of Go, Operating System, etc.): 1.20

[darccio]

Thanks @elijahj19 for the report. This seems to be related to #2518. Both have been unified as a single issue and fixed by #2521, where we included your test. Closing to track both issues in the earliest report.

[darccio]

Released as part of v1.60.1 cc @elijahj19