Improve field-profile/schema-resolution operator UX without weakening correctness

[shmsr]

Summary

The field-profile / schema-resolution engine is more honest than before, but the operator experience still asks users to understand too much internal vocabulary and diagnose too many failure modes manually.

This issue tracks the remaining product work after the schema review:

• correctness gaps we should not paper over with messaging
• operator UX gaps we can improve immediately
• the boundary between safer guidance and unsafe silent guessing

What is working today

• Grafana and Datadog both follow a plan -> emit -> verify model.
• Live _field_caps verification does not silently remap queries to another layout.
• Grafana auto detection is conservative and correct for named Prometheus layouts.
• Warnings / readiness contracts now surface more of the schema uncertainty than before.

Remaining operator problems

1. Operators think in ingest routes, not field-profile names

Today we expose:

• otel
• prometheus_native
• prometheus_remote_write
• prometheus_metrics
• elastic_agent

Operators usually think in terms like:

• Elasticsearch native Prometheus write
• Fleet / Agent Prometheus remote_write
• Metricbeat Prometheus
• Elastic Agent system metrics
• Datadog metrics moved to OTel
• I do not know; inspect the cluster

We should prefer an ingest-route-first UX and treat field-profile names as advanced/internal controls.

2. Translation success vs deployment readiness is still too easy to confuse

A migration can be structurally successful while still being wrong for the target schema.

We should present separate top-level verdicts for:

1. translation correctness
2. target schema readiness
3. live runtime/render readiness

3. Grafana -> ECS / Elastic-Agent targets remain a real capability gap

This is not just messaging.

Grafana migration is strongest when Prometheus metric names survive into Elasticsearch under a known layout. It is weaker when the target stores semantically renamed ECS / Elastic-Agent system fields instead of Prometheus names.

That path currently depends on explicit metric mapping / rule-pack knowledge rather than a first-class built-in target model.

4. Datadog metric-name translation is still too implicit

Datadog users naturally ask:

• are we assuming my target is already OTel-shaped?
• are host metrics different from custom app metrics?
• if Datadog Agent metrics move to OTel, do I need metric renames?
• are Datadog queries translated to PROMQL or ES|QL?

The real answers are:

• built-in otel mainly solves tag/attribute mapping, not Datadog metric-name -> OTel semconv translation
• built-in elastic_agent covers common system metrics, not arbitrary custom app metrics
• Datadog dashboards translate to Kibana / ES|QL-native queries, not to PROMQL

We should classify Datadog runs much more plainly:

• same metric names likely
• metric renames likely required
• custom app metrics likely need map

5. The tool still makes operators diagnose the failure class manually

When a migrated dashboard is empty, the operator needs to know whether the cause is:

• wrong field profile / wrong target layout
• missing telemetry
• metric-name drift
• tag / attribute drift
• unsupported translation semantics

We expose pieces of this today, but we do not yet compress them into a single clear diagnosis plus the exact next command to run.

Design constraints

We should improve UX without reducing honesty.

Good direction:

• stronger guidance
• better classification
• exact follow-up command suggestions

Bad direction:

• silently guessing semantic renames we cannot prove
• auto-switching layouts without operator confirmation
• reporting readiness from translation alone

Proposed work

Near-term UX work

• add ingest-route-first prompts / flags over raw field-profile terminology
• split top-level verdicts into translation vs schema readiness vs live readiness
• print exact follow-up commands whenever the tool has enough evidence
• classify Datadog runs by whether metric renames are likely

Medium-term product work

• first-class Grafana support for ECS / Elastic-Agent system-metric targets
• stronger per-panel field-resolution attribution
• stricter operator-safe mode that refuses to imply success on unverified schema assumptions

Source

This issue is extracted from docs/design/field-profile-schema-redesign.md in the branch work. That doc should be removed once this issue exists so the backlog lives in GitHub rather than in the PR branch.