observability.otlp.endpoint with a non-HTTPS URL silently crashes MCP Gateway startup — no compile-time validation or warning

[Etienne-M]

Compiler silently wires non-HTTPS observability.otlp.endpoint into MCP Gateway's HTTPS-only self-telemetry config, crashing gateway startup

Analysis

Symptom: A workflow using observability.otlp.endpoint pointed at a runner-local OTLP
collector (e.g. http://127.0.0.1:4318, the pattern used to capture the engine's own
OTLP spans — Copilot CLI/Claude tool/model calls — since gh-aw's built-in otel.jsonl
mirror only covers gh-aw's own JS-emitted spans) causes the "Start MCP Gateway" step to
fail with a generic wrapper error:

[error] ERROR: Gateway process (PID: 4168) exited during initialization
Error: Process completed with exit code 1.

The real cause is only visible in the gateway's own structured log
(mcp-logs/mcp-gateway.log, recoverable via gh aw audit <run-id> --json):

[ERROR] [startup] Configuration validation failed:
Configuration validation error (MCP Gateway version: v0.4.8):
  Location: /gateway/opentelemetry/endpoint
  Error: 'http://127.0.0.1:4318' does not match pattern '^(https://.+|\$\{[A-Z_][A-Z0-9_]*\})$'

Root cause (confirmed by reading source in this repo):

The compiler takes the single observability.otlp.endpoint value and feeds it into
two independent consumers with no validation that both can accept it:

1. pkg/workflow/observability_otlp.go — injectOTLPConfig() (around line 872) sets
   workflowData.OTLPEndpoint = firstEndpoint unconditionally, from whatever string the
   user configured. There is no HTTPS check anywhere in this file — confirmed by reading
   it in full; the only URL parsing (extractOTLPEndpointDomain,
   shouldRewriteAuthorizationForSentry) is for firewall-allowlisting and Sentry header
   rewriting, not scheme validation.
2. pkg/workflow/mcp_gateway_config.go — buildMCPGatewayConfig() (lines 206-207) passes
   that same value straight through: OTLPEndpoint: workflowData.OTLPEndpoint.
3. pkg/workflow/mcp_renderer.go (~line 266) renders it verbatim into the gateway's stdin
   JSON config as gateway.opentelemetry.endpoint.

This wiring was introduced in #24697 ("Configure MCP gateway OpenTelemetry from
observability.otlp and actions/setup trace IDs").

Why the gateway rejects it — confirmed correct, spec-compliant behavior, not a
gh-aw-mcpg bug: The HTTPS-only requirement is not a gh-aw-mcpg-side decision — it's
defined in this repo's own MCP Gateway Specification,
docs/src/content/docs/reference/mcp-gateway.md §4.1.3.7 "OpenTelemetry Configuration"
(current spec version 1.15.0): "endpoint MUST be an HTTPS URL", with the same rule
encoded in this repo's two canonical schema copies
(pkg/workflow/schemas/mcp-gateway-config.schema.json and
docs/public/schemas/mcp-gateway-config.schema.json, per the existing pattern
documented in docs/adr/32280-pass-otel-headers-as-container-env-var.md, which
describes these two files as "the source and published copies of the gateway config
schema"). github/gh-aw-mcpg is the reference implementation of this spec: its own
copy of the rule lives in internal/config/validation_tracing.go
(validateOpenTelemetryConfig) and internal/config/schema/mcp-gateway-config.schema.json
(note: this copy has already drifted textually from gh-aw's two copies — worth fixing
independently of this issue), enforced by dedicated compliance test T-OTEL-004
("Reject non-HTTPS endpoint"), introduced in gh-aw-mcpg#3188 closing gh-aw-mcpg#3187 to
implement this spec requirement. gh-aw-mcpg's own docs (docs/CONFIGURATION.md:630)
and its own dogfood workflow (.github/workflows/smoke-otel-tracing.md) only ever use
real HTTPS endpoints (Sentry ingest). There is no legacy http://-permitting escape
hatch reachable from JSON stdin config (the gateway.tracing legacy key that allows
http:// is TOML-only and gh-aw never emits TOML).

Two independent systems get conflated:

• network.allowed: local (AWF firewall) governs egress from the sandboxed agent
  container — correctly used here to let Copilot CLI itself reach the loopback
  collector for its own span export. This works fine and is unrelated to the crash.
• gateway.opentelemetry.* is the gateway process's own self-telemetry config,
  enforced by the gateway binary at startup before any network I/O. In workflows where
  the gateway container is started directly on the runner (before the sandboxed agent
  container even exists), it isn't inside the firewall sandbox at all, so
  network.allowed has no bearing on it.

The compiler has no compile-time detection, warning, or way to route these two
destinations independently today. Any workflow using a runner-local (plain-HTTP) OTLP
collector for the engine's own telemetry — otherwise a supported, documented pattern —
gets its MCP Gateway silently broken, with the real cause hidden behind a generic
"exited during initialization" message.

Note on prior related work: #25481 removed the observability-otlp.md import from
the smoke-claude/smoke-copilot workflows, but its PR description only says "per
@pelikhan's request" with no stated technical reason — I could not confirm it was
specifically about this HTTPS/loopback conflict, so treat it as circumstantial, not
confirmed precedent.

Reproduction

• Workflow frontmatter:

  observability:
    otlp:
      endpoint:
        - url: "http://127.0.0.1:4318"
  network:
    allowed:
      - defaults
      - local   # lets the sandboxed agent reach the loopback collector

  (matches the documented pattern for capturing engine-emitted OTLP spans via a
  runner-local otelcol-contrib collector)
• Compile with gh aw compile, run the workflow: the "Start MCP Gateway" step fails
  with the generic error above; mcp-logs/mcp-gateway.log in the run's artifacts has the
  real schema-validation error.

Implementation Plan

There are two ways to close this gap, and both are ultimately decisions for this repo
(gh-aw owns the MCP Gateway Specification and its schema; gh-aw-mcpg is the
reference implementation, maintained by the same team). Option A is the immediate,
self-contained fix and should land regardless of what happens with Option B — it's a
single-repo compiler change with no spec edit and no dependency on another release.
Option B is a spec-level relaxation: its first step (editing the spec text and
schema) is also a change in this repo, but it additionally requires a follow-up
implementation change in gh-aw-mcpg (validation code, its own schema copy, compliance
tests) and a new gateway image release before it takes effect — meaningfully more
coordination and lead time than Option A, not a different governance owner. It's
included here as a design option for whoever triages this issue to decide on; if
pursued, the gh-aw-mcpg implementation step should be filed as its own follow-up issue
once the spec change is actually decided/merged, rather than speculatively now.

Option A (primary — implement here)

1. pkg/workflow/observability_otlp.go / pkg/workflow/mcp_gateway_config.go: When
   the resolved OTLP endpoint is a statically known literal string (i.e. not a GitHub
   Actions expression — isGitHubActionsExpression already exists and is used for this
   exact distinction elsewhere in this file) and does not start with https://, do not
   propagate it into the MCP Gateway's config path
   (workflowData.OTLPEndpoint/OTLPHeaders as consumed by buildMCPGatewayConfig).
   Continue injecting OTEL_EXPORTER_OTLP_ENDPOINT etc. into the workflow-level env:
   block as today, so Copilot CLI's own span export (and any other consumer of the
   workflow env) is unaffected — only the gateway's gateway.opentelemetry block is
   skipped for this endpoint.
   • Endpoints that are GitHub Actions expressions (e.g. ${{ secrets.X }}) can't be
     validated at compile time; leave those wired through unchanged (the gateway's
     existing runtime validation still applies and produces a clear error if the
     resolved secret isn't HTTPS — this is existing, acceptable behavior, not part of
     this bug).
2. Emit a compiler warning (via the existing otlpLog/console warning machinery,
   similar in spirit to getOTLPIfMissingMode's warn/ignore handling) when this
   skip happens, explaining: "observability.otlp.endpoint '' is not HTTPS; skipping
   MCP Gateway self-telemetry (gateway.opentelemetry requires HTTPS per MCP Gateway Spec
   §4.1.3.7) — Copilot CLI's own OTLP export is unaffected." This turns a silent runtime
   crash into a visible, actionable compile-time signal.
3. Tests (pkg/workflow/observability_otlp_test.go,
   pkg/workflow/mcp_gateway_config_test.go or equivalent):
   • Literal http://127.0.0.1:4318 endpoint → workflowData.OTLPEndpoint used for the
     gateway config is empty / gateway.opentelemetry is omitted from rendered MCP
     config; OTEL_EXPORTER_OTLP_ENDPOINT env var is still injected with the original
     value.
   • Literal https://collector.example.com endpoint → unchanged existing behavior
     (wired into both).
   • ${{ secrets.GH_AW_OTEL_ENDPOINT }} expression endpoint → unchanged existing
     behavior (wired into both, deferred to runtime).
   • Compiler warning is emitted for the non-HTTPS literal case.
4. Docs: docs/src/content/docs/guides/open-telemetry.mdx (or reference page) —
   note that gateway.opentelemetry self-telemetry requires HTTPS and that runner-local
   collectors for engine-only spans won't be forwarded to the gateway's own tracing.
5. Run make agent-finish per repo conventions.

Option B (spec-level relaxation — larger, multi-repo change)

Relax the MCP Gateway Specification itself so gateway.opentelemetry.endpoint
tolerates plain http:// when the host resolves to a loopback address
(127.0.0.0/8, ::1, localhost) — analogous to RFC 8252's loopback exception for
OAuth native-app redirect URIs, since loopback traffic never traverses a real network
and so has no interception surface. If pursued, it would make the runner-local-collector
pattern work against the gateway's own telemetry too, without gh-aw needing to
special-case anything in the compiler.

Steps, if the team decides to pursue this:

1. In this repo: amend §4.1.3.7 of
   docs/src/content/docs/reference/mcp-gateway.md to state the loopback exception,
   bump the spec version per this project's spec process, and update both schema copies
   (pkg/workflow/schemas/mcp-gateway-config.schema.json,
   docs/public/schemas/mcp-gateway-config.schema.json) to relax the endpoint
   pattern accordingly (keeping the HTTPS requirement for all non-loopback hosts).
2. Follow-up in github/gh-aw-mcpg (file once step 1 is merged, not speculatively
   now): update internal/config/validation_tracing.go
   (validateOpenTelemetryConfig) and its own copy of
   internal/config/schema/mcp-gateway-config.schema.json to match, add a
   loopback-acceptance test alongside the existing T-OTEL-004 ("Reject non-HTTPS
   endpoint") test, and cut a new gateway image release.

This is not included in Option A's scope because it's a larger, slower-landing
change — it touches a versioned spec/schema most other MCP Gateway config fields also
rely on, and only takes effect after a new gateway image is built and released —
whereas Option A ships in a single gh-aw release with no spec change at all. Both can
be pursued; Option A should not wait on Option B.

Why Option A can land immediately, independent of Option B

The HTTPS-only rule is this repo's own documented, versioned specification requirement
(§4.1.3.7, T-OTEL-004), correctly implemented by gh-aw-mcpg — verified this is
intentional, not a regression, by reading the spec, both schema copies, and
gh-aw-mcpg's validation code, its schema, docs, and tests. Regardless of whether
Option B's spec relaxation is ever pursued, the compiler shouldn't silently hand the
gateway a config that's already known (at compile time, for literal endpoints) to
violate that requirement — that's Option A, and it can ship on its own.