openkruise · Liquorice-Ma · Jul 10, 2026
diff --git a/docs/proposals/20260709-request-id-as-traceid.md b/docs/proposals/20260709-request-id-as-traceid.md
@@ -0,0 +1,193 @@
+---
+title: Request ID as TraceID for Unified Correlation
+authors:
+  - "@mahe"
+reviewers:
+  - "@TBD"
+creation-date: 2026-07-09
+last-updated: 2026-07-09
+status: provisional
+see-also:
+  - "/docs/proposals/20260702-sandbox-otel-distributed-tracing.md"
+---
+
+# Request ID as TraceID for Unified Correlation
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals / Future Work](#non-goals--future-work)
+- [Design](#design)
+  - [Core Changes](#core-changes)
+  - [Root Span Creation Location Change](#root-span-creation-location-change)
+  - [Request ID as TraceID](#request-id-as-traceid)
+  - [OTel Non-Blocking Async Export](#otel-non-blocking-async-export)
+  - [Controller Span Noise Optimization](#controller-span-noise-optimization)
+  - [Unified Trace-Log Correlation](#unified-trace-log-correlation)
+  - [Unchanged Parts](#unchanged-parts)
+- [Risks and Mitigations](#risks-and-mitigations)
+- [Alternatives](#alternatives)
+- [Implementation History](#implementation-history)
+
+## Overview
+
+This proposal modifies the trace context generation scheme from
+[20260702-sandbox-otel-distributed-tracing.md](./20260702-sandbox-otel-distributed-tracing.md)
+with four changes:
+
+1. **Root Span creation moved up**: from individual operation functions in `api.go`
+   (ClaimSandbox, DeleteSandbox, etc.) to the HTTP middleware layer (`framework.go`),
+   unifying trace lifecycle management.
+2. **Request ID as TraceID**: use the existing HTTP request ID (UUID) directly as the
+   OTel TraceID, enabling unified trace-log correlation. Implemented via a custom
+   `IDGenerator`, avoiding manual span context construction.
+3. **OTel non-blocking async export**: confirmed `BatchSpanProcessor` async batch export
+   mechanism; enabling tracing does not affect business performance.
+4. **Controller span noise optimization**: move `StartReconcileSpan` after the
+   `shouldRequeue` check, avoiding meaningless spans for μs-level query-only Reconciles.
+
+## Motivation
+
+### Current Problems
+
+1. **Trace-log disconnect**: In the 0702 proposal, TraceID is auto-generated by the OTel SDK,
+   with no direct mapping to the `requestID` field in logs. Troubleshooting requires finding
+   the requestID in logs first, then searching for the corresponding trace in Jaeger —
+   cannot be done with a single ID.
+2. **Scattered Root Span creation**: `WithRootSpanContext(ctx)` is scattered across 5
+   operation functions in `api.go` (Claim, Clone, Pause, Resume, Delete). Every new operation
+   requires manually adding a line, prone to omission.
+3. **Middleware execution time not covered**: Root Span is created inside operation functions,
+   not including middleware execution time (e.g., CheckApiKey).
+
+### Goals
+
+1. **One ID for both trace and logs**: request ID doubles as TraceID; searching request ID
+   in Jaeger finds the complete trace, and logs carry the same request ID.
+2. **Unified Root Span creation**: auto-created at the middleware layer; operation functions
+   no longer need to manually call `WithRootSpanContext`.
+3. **Full request lifecycle coverage**: Root Span covers middleware + handler execution time.
+
+### Non-Goals / Future Work
+
+- Do not change controller-side trace context extraction logic (still propagates W3C
+  traceparent via annotation)
+- Do not change OTel Collector / Jaeger deployment architecture
+- Do not inject TraceID field into logs (request ID correlation is sufficient)
+
+## Design
+
+### Core Changes
+
+| Dimension | 0702 Scheme | 0709 Scheme |
+|-----------|------------|------------|
+| TraceID generation | OTel SDK auto-generated | Request ID (UUID without hyphens) as TraceID |
+| Root Span location | `api.go` per-operation functions | `framework.go` HTTP middleware |
+| `WithRootSpanContext` | Manual call in each operation | Single call in middleware |
+| Trace-log correlation | No direct correlation | TraceID = Request ID, direct search |
+
+### Root Span Creation Location Change
+
+**Before** (0702 scheme): Each operation function (`api.go`) manually calls
+`WithRootSpanContext(ctx)`. Root Span is created inside the operation function, not covering
+middleware execution time. Every new operation requires manually adding the call.
+
+**After** (0709 scheme): Root Span is created uniformly in the `framework.go` middleware layer,
+covering the full request lifecycle (middleware + handler). Operation functions (`api.go`)
+remove all `WithRootSpanContext` calls, keeping only business sub-span creation.
+
+### Request ID as TraceID
+
+UUID (v4) without hyphens yields 32 hex characters = 16 bytes, exactly matching the OTel
+TraceID format.
+
+**Implementation**: Implement the OTel SDK's `IDGenerator` interface
+(`pkg/tracing/idgenerator.go`). When creating the Root Span, the TraceID is read from the
+request ID stored in the context. The middleware stores the request ID via
+`WithRequestID(ctx, requestID)`, and the SDK automatically calls
+`IDGenerator.NewIDs(ctx)` to generate the TraceID.
+
+**Why not manually construct span context**: Manually constructing a span context causes Jaeger
+to report "missing parent span" (the constructed span context is treated as the root span's
+parent, but that parent span does not exist). A custom `IDGenerator` lets the SDK generate
+the TraceID internally, so the root span has no parent, avoiding this issue.
+
+**Fallback**: When the request ID is not a standard UUID format, the `IDGenerator` falls back
+to random TraceID generation. The span still carries a `request.id` attribute for searchability.
+
+### OTel Non-Blocking Async Export
+
+Enabling tracing does not block or slow down business logic:
+
+1. **Async batch export**: `provider.go` uses `BatchSpanProcessor`. `span.End()` only enqueues
+   span data into an in-memory queue and returns immediately. A background goroutine
+   periodically batches and sends to the OTLP gRPC exporter.
+2. **Minimal overhead**: `tracer.Start()` / `span.End()` involve only memory operations,
+   no I/O blocking.
+3. **Auto-degradation**: When the OTel Collector is unavailable, span data accumulates in the
+   queue and is eventually dropped, without affecting business functionality.
+4. **Controlled memory**:实测 manager Pod memory ~28Mi, well below threshold.
+
+### Controller Span Noise Optimization
+
+#### Problem
+
+The controller's Reconcile has many `Ensure*` sub-spans, a large number of which are pure
+query operations (e.g., checking Pod status, calculating new status) with execution times of
+only tens of μs. These spans appear as noise in Jaeger.
+
+#### Solution
+
+Move `StartReconcileSpan` after `calculateStatus` + `shouldRequeue` check. Reconciles with
+`shouldRequeue=true` (pure status updates, no Pod operations) do not create spans. The
+terminating branch (delete operations) creates spans separately, as `EnsureSandboxTerminated`
+sub-spans require a Reconcile span as parent.
+
+#### Effect
+
+- Remove μs-level query-only Reconcile spans
+- Retain Reconcile spans with actual Pod operations (Pending → Running, Running → Paused, etc.)
+
+### Unified Trace-Log Correlation
+
+Troubleshooting workflow:
+1. Get `requestID` from logs
+2. Search for that request ID in Jaeger (it is the TraceID)
+3. View the complete manager → controller call chain
+
+### Unchanged Parts
+
+- **Annotation propagation**: still uses `agents.kruise.io/trace-context` to inject W3C traceparent
+- **Controller-side extraction**: `ExtractTraceContext` and `StartReconcileSpan` unchanged
+- **OTel SDK config**: TracerProvider, BatchSpanProcessor, OTLP gRPC exporter unchanged
+
+## Risks and Mitigations
+
+| Risk | Mitigation |
+|------|-----------|
+| Non-standard request ID causes TraceID mismatch | Fallback to OTel auto-generation; span still carries `request.id` attribute for search |
+| UUID v4 byte order mismatch with OTel TraceID | Both are 128-bit random values; byte order is irrelevant for unique identification |
+| Hardcoded `sandbox-manager` tracer name in middleware | Acceptable: framework.go already depends on sandbox-manager/logs package |
+
+## Alternatives
+
+1. **Keep OTel auto TraceID + request ID as span attribute**: don't change TraceID generation,
+   only add `request.id` attribute to spans. Drawback: TraceID and request ID are different
+   values, requiring mapping during troubleshooting — not direct enough.
+
+2. **No traceparent, use request ID attribute only**: manager and controller are independent
+   traces, correlated via tag search. Drawback: cannot see the complete call chain hierarchy
+   in Jaeger.
+
+This proposal chose **request ID directly as TraceID**, balancing trace correlation (same
+TraceID) and log correlation (TraceID = request ID), with minimal changes and optimal effect.
+
+## Implementation History
+
+- [x] 07/09/2026: Proposed, modified framework.go and api.go, compilation verified
+- [x] 07/09/2026: Implemented custom `IDGenerator`, resolved "missing parent span" issue
+- [x] 07/09/2026: Verified TraceID = request ID in Jaeger (create operation, 5 spans, no Incomplete)
+- [x] 07/09/2026: Confirmed OTel BatchSpanProcessor async non-blocking, manager memory ~28Mi
+- [ ] 07/10/2026: Controller span noise optimization (move `StartReconcileSpan`)