fix(datadog): readable free-board layout + chrome polish

docs/superpowers/specs/2026-07-21-datadog-free-board-kibana-layout-design.md

@@ -0,0 +1,77 @@
+# Datadog free-board → Kibana layout (column-band)
+
+**Date:** 2026-07-21  
+**Status:** Implemented (hybrid C) — iterate on packing/band heuristics as needed  
+**Scope:** Wide Datadog `layout_type: free` dashboards (canvas extent ≫ 12), e.g. HAProxy, Apache, nginx-ingress
+
+## Problem
+
+Datadog free boards use a fine canvas (often 100–200 units wide). Mapping that onto Kibana’s 48-column grid with a single linear scale produces unreadable tiles (metrics/charts ~4 cols). Inflating type min-widths without a column model then overlaps neighbors; horizontal “fixes” that move `x` scramble column alignment (HAProxy looked broken).
+
+## Goal
+
+Optimize for **Kibana readability** while preserving the source board’s **left→right column story**:
+
+- Do **not** merge distinct source columns.
+- Do **not** reorder bands L→R.
+- Do give each band usable Kibana widths (chart ≥ 8, metric ≥ 6, markdown ≥ 4 when space allows).
+- No overlapping panels after layout.
+
+Ordered / ~12-column Datadog boards keep the existing row/heuristic path unchanged.
+
+## Approach (chosen): column-band layout
+
+1. **Detect bands** — Cluster leaf panels by Datadog `_dd_x` starts. Gap threshold ≈ half the median widget width (with a small absolute floor). Sort bands left→right.
+2. **Assign band widths** — Weight each band by the max `_dd_w` of panels whose *primary* band is that band (or by band span for multi-band widgets). Normalize weights to 48 columns. Apply readable floors per band based on the heaviest panel family in that band. If floors exceed 48, scale all band widths down proportionally but keep every band ≥ 1 and prefer cutting markdown/note slack before chart/metric floors.
+3. **Place panels**
+   - Primary band = band containing `_dd_x` (or leftmost overlapping band).
+   - Width = sum of widths of bands the panel’s `[x, x+w)` covers (span). Single-band panels fill their band width (or a sub-slot when multiple non-overlapping siblings share a horizontal slice — see below).
+   - `x` = start of the leftmost covered band.
+   - Height = `max(type min_h, round(_dd_h * scale_y))` with `scale_y` derived from a stable vertical scale (same global scale factor as today, or band-local only for packing anchors). Placeholder markdown may grow from content height.
+4. **Sub-slots in a band** — When two+ panels share the same band and overlapping y-range but different `_dd_x` within that band (e.g. HAProxy KPIs at x=25 and x=43 inside the overview band), split the band width proportionally by their `_dd_w` without creating new top-level bands. This densifies KPIs *inside* a column without merging columns.
+5. **Vertical pack** — Existing column-aware packer: place by `(anchor_y, x)`; drop only when x-ranges overlap. Do **not** run the pairwise overlap pusher on free boards.
+6. **Normalize** — Height mins/max only at the free-board path (no type min-width expansion that escapes the band model). Strip private `_free_*` / `_dd_*` keys as today.
+
+## Non-goals
+
+- Pixel-perfect Datadog fidelity.
+- Reflowing KPIs into new visual columns or changing L→R order.
+- YAML-only schema work (native IR / existing generate path is enough).
+- Fixing Datadog image URL placeholders or `check_status` semantics (separate).
+
+## Success criteria
+
+| Check | Pass rule |
+|---|---|
+| Column stability | Panels that share a source `_dd_x` keep the same Kibana `x` after layout |
+| Readable charts | Every `line`/`area`/`bar`/table free-board panel `w ≥ 8` when board has ≤ 6 bands; if more bands, best-effort with proportional shrink |
+| Readable metrics | Every metric/gauge panel `w ≥ 6` under the same band-count caveat |
+| No overlaps | Zero AABB overlaps after layout + normalize + pack |
+| Regression | Existing ordered-board / interleaved-note tests still pass |
+| Live | Remigrate HAProxy + Apache (+ nginx-ingress if available); upload to local Kibana; screenshot top of board shows distinct readable columns |
+
+## Test plan
+
+**Unit (generate.py / TestYAMLGeneration):**
+
+1. Keep `test_wide_free_board_keeps_global_column_scale` (aligned x across rows).
+2. Keep/extend `test_haproxy_style_free_board_keeps_dense_columns` — assert metric `w ≥ 6`, chart `w ≥ 8`, log column on the right, KPIs near top.
+3. Add Apache-style fixture: multiple mid-board columns; assert no overlaps and chart mins.
+4. Add band-detection unit tests: known `_dd_x` set → expected band count/order.
+5. Ordered Redis-style interleaved note test unchanged.
+
+**Integration:**
+
+- Migrate `haproxy.json`, `apache.json`, `nginx-ingress-controller.json` with upload to local stack.
+- Assert layout validation pass; spot-check YAML positions; full-page screenshots under `popular_ui_controls_20260721/local_ui/`.
+
+## Files
+
+- `observability_migration/adapters/source/datadog/generate.py` — replace `_apply_free_board_layout` (+ helpers for band cluster/assign/sub-slot).
+- `tests/test_datadog_migrate.py` — layout assertions above.
+
+## Risks
+
+- Over-clustering (too many bands) → still cramped; mitigate with gap threshold tuning and sub-slots.
+- Under-clustering (merge overview+frontend) → wrong story; mitigate by using x-start clusters, not centers, and not merging across large gaps.
+- Spanning notes (`width` covering many columns) must sum band widths correctly or they look short.

observability_migration/adapters/source/datadog/display.py

@@ -9,6 +9,7 @@
 
 from __future__ import annotations
 
+import re
 from typing import Any
 
 from .models import NormalizedWidget, TranslationResult
@@ -68,6 +69,14 @@ def enrich_panel_display(
         return yaml_panel
 
     unit_format = _resolve_unit(widget)
+    if not unit_format:
+        unit_format = _default_number_format(widget)
+    # Count-like KPIs (connections, reloads, …) should render as whole numbers
+    # even when Datadog left precision=2 — ``12.00`` reads as noisy in Kibana.
+    if unit_format and _looks_like_count_metric(widget):
+        unit_format = dict(unit_format)
+        unit_format["decimals"] = 0
+        unit_format["compact"] = True
     if unit_format:
         _apply_format(esql, unit_format, result.kibana_type)
 
@@ -318,9 +327,57 @@ def _resolve_unit(widget: NormalizedWidget) -> dict[str, Any] | None:
     fmt = dict(fmt)
     if widget.precision is not None and widget.precision >= 0:
         fmt["decimals"] = widget.precision
+    elif "decimals" not in fmt and fmt.get("type") in {"number", "bytes", "bits"}:
+        # Prefer compact whole numbers when Datadog didn't set precision.
+        if _looks_like_count_metric(widget):
+            fmt["decimals"] = 0
+            fmt["compact"] = True
     return fmt
 
 
+def _default_number_format(widget: NormalizedWidget) -> dict[str, Any]:
+    """Sane Kibana defaults when Datadog left unit/precision unset."""
+    if widget.precision is not None and widget.precision >= 0:
+        return {"type": "number", "decimals": widget.precision, "compact": True}
+    if _looks_like_count_metric(widget):
+        return {"type": "number", "decimals": 0, "compact": True}
+    if _looks_like_percent_metric(widget):
+        return {"type": "number", "suffix": "%", "decimals": 1}
+    return {"type": "number", "decimals": 1, "compact": True}
+
+
+def _looks_like_percent_metric(widget: NormalizedWidget) -> bool:
+    title = str(widget.title or "")
+    if "%" in title:
+        return True
+    lowered = title.lower()
+    return "percent" in lowered or "pct" in lowered or "2xx" in lowered or "5xx" in lowered
+
+
+def _looks_like_count_metric(widget: NormalizedWidget) -> bool:
+    """Heuristic: connection/request/count tiles should not show .000 decimals."""
+    if _looks_like_percent_metric(widget):
+        return False
+    title = str(widget.title or "").lower()
+    count_words = (
+        "count", "connection", "reload", "request", "error", "session",
+        "server", "client", "hit", "miss", "event", "message",
+    )
+    if any(word in title for word in count_words):
+        return True
+    for query in widget.queries or []:
+        mq = getattr(query, "metric_query", None)
+        if mq is None:
+            continue
+        agg = str(getattr(mq, "aggregation", "") or "").lower()
+        metric = str(getattr(mq, "metric", "") or getattr(mq, "name", "") or "").lower()
+        if agg in {"count", "cardinality"}:
+            return True
+        if agg == "sum" and any(tok in metric for tok in ("count", "connection", "request", "reload")):
+            return True
+    return False
+
+
 def _apply_format(
     esql: dict[str, Any],
     fmt: dict[str, Any],
@@ -365,12 +422,14 @@ def _apply_legend(
         esql.setdefault("legend", {
             "visible": "show" if shown else "hide",
             "position": "right",
-            "truncate_labels": 1,
+            # 0 = disable truncation (schema); "1" clipped pod names to
+            # ``controller_p…`` on breakdown charts.
+            "truncate_labels": 0,
         })
     elif kibana_type in ("partition", "treemap"):
         esql.setdefault("legend", {
             "visible": "auto" if shown else "hide",
-            "truncate_labels": 1,
+            "truncate_labels": 0,
         })
     elif kibana_type == "heatmap":
         appearance = esql.setdefault("appearance", {})
@@ -450,8 +509,18 @@ def _apply_axis(yaml_panel: dict[str, Any], widget: NormalizedWidget, result: Tr
 
 
 def _clean_template_vars(title: str) -> str:
-    """Replace Datadog template variable placeholders for Kibana."""
-    import re
-    title = re.sub(r"\$(\w+)\.value", r"{\1}", title)
-    title = re.sub(r"\$(\w+)", r"{\1}", title)
-    return title
+    """Strip Datadog ``$template`` placeholders from panel titles.
+
+    Datadog often appends ``over $host`` (and friends). Kibana already has
+    matching Options List controls, so leaving ``$host`` or ``{host}`` in the
+    chrome reads as a migration bug.
+    """
+    cleaned = re.sub(
+        r"\s+over\s+(\$[\w.]+)(\s*,\s*\$[\w.]+)*\s*$",
+        "",
+        title or "",
+        flags=re.IGNORECASE,
+    )
+    cleaned = re.sub(r"\$[\w.]+", "", cleaned)
+    cleaned = re.sub(r"\{[\w.]+\}", "", cleaned)
+    return re.sub(r"\s{2,}", " ", cleaned).strip(" ,-|")

observability_migration/adapters/source/datadog/field_map.py

@@ -21,6 +21,7 @@
 
 import yaml
 
+from observability_migration.core.kibana_safe_fields import kibana_safe_field_name
 from observability_migration.core.verification.field_capabilities import (
     FieldCapability,
     fetch_field_capabilities,
@@ -78,13 +79,23 @@ def map_tag(self, dd_tag: str, context: str = "") -> str:
         if dd_tag in tag_map:
             mapped = tag_map[dd_tag]
             if context == "metric" and mapped in _LOG_ONLY_FIELDS:
-                return dd_tag
-            return self._prefer_aggregatable_keyword_subfield(mapped, context=context)
+                return kibana_safe_field_name(dd_tag)
+            return kibana_safe_field_name(
+                self._prefer_aggregatable_keyword_subfield(mapped, context=context)
+            )
         if context == "log" and self.log_tag_map:
-            return self._prefer_aggregatable_keyword_subfield(dd_tag, context=context)
+            return kibana_safe_field_name(
+                self._prefer_aggregatable_keyword_subfield(dd_tag, context=context)
+            )
         if self.tag_prefix:
-            return self._prefer_aggregatable_keyword_subfield(f"{self.tag_prefix}{dd_tag}", context=context)
-        return self._prefer_aggregatable_keyword_subfield(dd_tag, context=context)
+            return kibana_safe_field_name(
+                self._prefer_aggregatable_keyword_subfield(
+                    f"{self.tag_prefix}{dd_tag}", context=context
+                )
+            )
+        return kibana_safe_field_name(
+            self._prefer_aggregatable_keyword_subfield(dd_tag, context=context)
+        )
 
     def _prefer_aggregatable_keyword_subfield(self, field_name: str, context: str = "") -> str:
         """Prefer ``field.keyword`` when live caps show the base field is unsafe for grouping."""
@@ -204,6 +215,8 @@ def _default_tag_map() -> dict[str, str]:
         "availability_zone": "cloud.availability_zone",
         "zone": "cloud.availability_zone",
         "status": "log.level",
+        # Bare ``key`` breaks Kibana Options List field lookup; use dotted label.
+        "key": "labels.key",
         "container_name": "container.name",
         "container_id": "container.id",
         "@http.url_details.path": "http.url",