How to force sampling when creating a span

[idanya]

I am trying to force sampling by a request header.
For a sampler, I'm using ParentBased with TraceIDRatioBased:

sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(config.SampleRate)))

To start a new span (upon HTTP request), I call Start:

tracer.Start(ctx, spanName)

So in order to force a sampling, I want to check the request headers, and based on my logic decide if I should sample that request (regardless and on top of the ratio based sampling). To do that, I am wrapping the request context like so:

spanContext := trace.NewSpanContext(trace.SpanContextConfig{})
wrappedContext := trace.ContextWithSpanContext(ctx,
		spanContext.WithTraceFlags(spanContext.TraceFlags().WithSampled(true)))

I do get a context with span config that is sampled, but when Start is called, the span context that is returned is not sampled anymore, or at least it's span.SpanContext().IsSampled() returns false

What am I missing?
Is there another way to force sampling and achieve my requirement? Seems to me it's a simple and widely used use case.

[MrAlias]

Your spanContext contains invalid trace and span IDs (0).

opentelemetry-go/trace/trace.go

Lines 54 to 58 in f4ec95d

	// IsValid checks whether the trace TraceID is valid. A valid trace ID does
	// not consist of zeros only.
	func (t TraceID) IsValid() bool {
	return !bytes.Equal(t[:], nilTraceID[:])
	}

opentelemetry-go/trace/trace.go

Lines 77 to 81 in f4ec95d

	// IsValid checks whether the SpanID is valid. A valid SpanID does not consist
	// of zeros only.
	func (s SpanID) IsValid() bool {
	return !bytes.Equal(s[:], nilSpanID[:])
	}

This means the SpanContext itself is considered invalid.

opentelemetry-go/trace/trace.go

Lines 215 to 219 in f4ec95d

	// IsValid returns if the SpanContext is valid. A valid span context has a
	// valid TraceID and SpanID.
	func (sc SpanContext) IsValid() bool {
	return sc.HasTraceID() && sc.HasSpanID()
	}

opentelemetry-go/trace/trace.go

Lines 242 to 245 in f4ec95d

	// HasTraceID checks if the SpanContext has a valid TraceID.
	func (sc SpanContext) HasTraceID() bool {
	return sc.traceID.IsValid()
	}

opentelemetry-go/trace/trace.go

Lines 263 to 266 in f4ec95d

	// HasSpanID checks if the SpanContext has a valid SpanID.
	func (sc SpanContext) HasSpanID() bool {
	return sc.spanID.IsValid()
	}

Because of this the parent based sampler will use the root sampler to make a sampling decision for you child span.

opentelemetry-go/sdk/trace/sampling.go

Lines 265 to 281 in f4ec95d

	func (pb parentBased) ShouldSample(p SamplingParameters) SamplingResult {
	psc := trace.SpanContextFromContext(p.ParentContext)
	if psc.IsValid() {
	if psc.IsRemote() {
	if psc.IsSampled() {
	return pb.config.remoteParentSampled.ShouldSample(p)
	}
	return pb.config.remoteParentNotSampled.ShouldSample(p)
	}
	if psc.IsSampled() {
	return pb.config.localParentSampled.ShouldSample(p)
	}
	return pb.config.localParentNotSampled.ShouldSample(p)
	}
	return pb.root.ShouldSample(p)
	}

This means your child span will be sampled at the rate you configured for ratio based root sampler.

To always sample the child span, make sure the parent span context contains valid trace and span identifiers and that the WithLocalParentSampled option is set to AlwaysSample (this is the default).

[MrAlias]

Why not use a Tracers Start method? That indeed seems like the correct approach to creating a span.

[idanya]

To my understanding, in order to force sampling on a specific span (and its children), I have to set the TraceFlags. This is not something I can do with Start(). For that reason, I got into creating the span manually in the first place.

[MrAlias]

To force spans to be sampled use an appropriate Sampler. For example, AlwaysSample, or ParentBased with and AlwaysSample for its root.

Forcing single spans to be sampled outside of the context of a trace is contrary to the point of tracing. Complete traces are what provide context and meaningful information. If you are looking for metrics about a single operation and trying to capture those in a span you would be better suited to do so with dedicated metrics.

[idanya]

I am trying to force the root span, not a single out-of-context one.
The root span is starting upon an HTTP request in my case.
What I am trying to achieve is having this root span and all of its children sampled in some cases despite the fact that it might not, according to the RatioBased sampler.

[MrAlias]

If you want to force sampling based on some application code for an entire trace you can do this with a custom sampler. For example:

package main

import (
	"context"
	"fmt"

	"go.opentelemetry.io/otel/sdk/trace"
	traceapi "go.opentelemetry.io/otel/trace"
)

type key int

const sampleKey key = iota

type sampler struct {
	delegate trace.Sampler
}

func New(delegate trace.Sampler) trace.Sampler {
	return &sampler{delegate: delegate}
}

func (s *sampler) ShouldSample(p trace.SamplingParameters) trace.SamplingResult {
	if _, ok := p.ParentContext.Value(sampleKey).(struct{}); ok {
		return trace.SamplingResult{
			Decision:   trace.RecordAndSample,
			Tracestate: traceapi.SpanContextFromContext(p.ParentContext).TraceState(),
		}
	}
	return s.delegate.ShouldSample(p)
}

func (s *sampler) Description() string {
	return fmt.Sprintf("my-sampler{%s}", s.delegate.Description())
}

func main() {
	s := New(trace.ParentBased(trace.TraceIDRatioBased(0.5)))
	tracer := trace.NewTracerProvider(trace.WithSampler(s)).Tracer("my-app")

	ctx := context.WithValue(context.Background(), sampleKey, struct{}{})
	_, span := tracer.Start(ctx, "span name")
	fmt.Println(span.SpanContext().IsSampled())
}

You can also restrict this custom sampler to only local/remote and sampled/unsampled spans of the ParentBased sampler (1, 2, 3, 4) if you wanted to scope this more narrowly.

[dmathieu]

I think the problem is that the nonRecordingSpan created when calling ContextWithSpanContext has no TraceID nor SpanID set, because none were passed to SpanContextConfig.

I was able to get your example to work with the following:

package main

import (
	"context"
	"log"

	"go.opentelemetry.io/otel"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

const (
	sampleRatio  = 0
	shouldSample = true
)

func main() {
	sampler := sdktrace.ParentBased(sdktrace.TraceIDRatioBased(sampleRatio))
	tp := sdktrace.NewTracerProvider(sdktrace.WithSampler(sampler))
	otel.SetTracerProvider(tp)

	tracer := otel.Tracer("")

	spanContext := trace.NewSpanContext(trace.SpanContextConfig{
		TraceID: [16]byte{1}, // TODO: randomly generate
		SpanID:  [8]byte{1}, // TODO: randomly generate
	})
	ctx := trace.ContextWithSpanContext(context.Background(),
		spanContext.WithTraceFlags(spanContext.TraceFlags().WithSampled(true)))

	ctx, span := tracer.Start(ctx, "span")

	log.Printf("Span IsSampled: %#v", span.SpanContext().IsSampled())
	defer span.End()

	_, span2 := tracer.Start(ctx, "span2")
	log.Printf("Span2 IsSampled: %#v", span.SpanContext().IsSampled())
	defer span2.End()
}

Of course, manually having to set those TraceID and SpanID is quite cumbersome. A nicer way of generating that SpanContext with custom flags would be interesting.

[idanya]

Exactly, this is why it may be worth exposing randomIDGenerator as I mentioned.

[dmathieu]

Or something such as trace.NewSpanContextWithDefaults()

[dmathieu]

Another idea, which would avoid changing the public API would be to always make sure the created nonRecordingSpan has a valid SpanContext by setting a TraceID and SpanID in ContextWithSpanContext if they are null.