gfx/plate-sprint: OUTDOOR-LORA trained + smoke tested (#1481) — camp borderline, forest_road regresses

qa/evidence/plate-sprint/outdoor-lora/SMOKE-TEST-VERDICT.md

@@ -0,0 +1,68 @@
+# OUTDOOR-LORA smoke test verdict (2026-07-10)
+
+Model: `model_RsWEcQL2NWXwoyEodWVE2vWG` ("WorldOS Painterly Exterior (FLUX)"), trained on the
+18-image quality-passed set in `training_manifest.md` (job `job_U3HDsSj7T4aPETy7MRCuX6oK`,
+1080 CU, ~172 min).
+
+Pipeline per room: mint anchor via schema-correct `model_run` (`model_bfl-flux-1-dev` + depth
+ControlNet str 0.65/end 0.6 + the new LoRA, `lorasScale` 0.7) -> Gemini instruction-edit
+re-registration (`model_google-gemini-3-1-flash`, structure-lock + dimetric-lock prompt,
+`referenceImages=[minted anchor]`) over the room's own greybox -> overlay + advisory edge-recall
+(`qa/plate_overlays.py` / `advisory_eval.py`, content-blind per #1491) -> 5-scorer blind panel
+(control + reference/incumbent + candidate, shuffled, `qa/scores_db.py` surface=visual).
+
+## Results
+
+| Room | Candidate median | Reference | Baseline cap | Success bar | Result |
+|---|---|---|---|---|---|
+| forest_road | **2.0 / 10** | cross-lane anchor (camp_clearing_night_v2) 7.0, control 9.0 | 6.0 | >6.5 | **SEVERE REGRESSION** |
+| camp_clearing_night | **6.5 / 10** | incumbent (camp_clearing_night_v2, actual adopted plate) 6.0, control 9.0 | 6.0 | >6.5 | **BORDERLINE** (right at the line, not clearly above) |
+
+Full scores, mapping, and defect notes: `forest_road/panel_verdict.json`, `camp/panel_verdict.json`.
+scores_db rows: `outdoor-lora-smoke-forest_road-2026-07-10`, `outdoor-lora-smoke-camp-2026-07-10`.
+
+## Headline finding
+
+The new exterior-only, quality-passed LoRA **does not uniformly fix the outdoor generalization
+gap** the effort was built to close (issue #1481). It generalizes cleanly to **camp**
+(no invented architecture, no characters, modest +0.5 median lift over the incumbent — though
+short of a clean pass) but **regresses severely on forest_road**: all 5 blind scorers
+independently flagged broken/wireframe/"melted" architecture — the greybox's dense tree-line box
+volumes got repainted as disconnected wooden shrine/hut structures with carved doors and
+curtains, not trees. This is the SAME invented-architecture failure mode ARM C (the interior
+LoRA) produced on this room (CB-FOREST precedent, median 6.0 there) — just worse (median 2.0)
+and with a different visual signature (broken wireframe vs solid stone).
+
+**Root-cause hypothesis (new, beyond the original training-data-domain framing):**
+forest_road's greybox represents its tree line as adjacent axis-aligned rectangular box volumes
+(`forest_road_greybox.png` / `forest_road_greybox_depth.png`) — a coarser, more literally
+"architectural-looking" placeholder shape than camp's greybox. The model appears to read blocky
+rectangular volumes as buildings regardless of whether its LoRA was trained on interior or
+exterior images — a **geometry-shape bias**, not purely a training-domain bias. Edge-recall
+(the automated advisory metric) is misleading here: forest_road scored a deceptively HIGH 0.9678
+specifically because the candidate preserved the box edges too literally (as walls, not trees),
+while camp's genuinely better result scored a LOWER 0.6645 because it correctly reinterpreted
+the boxes as rounded organic forms. The overlay images are the binding evidence, not the number
+(confirmed both visually and by the 5-scorer panel, independent of the metric).
+
+## What this means for issue #1481
+
+- The OUTDOOR-LORA is a genuine, validated improvement for **camp-class** rooms (open clearings
+  with sparse, discrete tree/prop placeholders) but is **not ready to replace** ARM C or be
+  declared the fix for forest_road-class rooms (dense linear tree corridors with adjacent box
+  placeholders) without either (a) a differently-authored greybox for that structure class
+  (more organic/clustered tree footprints instead of axis-aligned boxes), or (b) further LoRA/
+  prompt iteration specifically targeting box-shaped placeholder reinterpretation.
+- Do not adopt either candidate as a new canonical_plate off this single smoke test. camp's
+  result is a borderline-promising lead worth one more iteration (fix the firepit ring artifact);
+  forest_road's result should be treated as a confirmed non-fix.
+
+## Cost actuals (record on #1481)
+
+- Training: 1080 CU (job_U3HDsSj7T4aPETy7MRCuX6oK, ~172 min, model_RsWEcQL2NWXwoyEodWVE2vWG)
+- Smoke test: 4 anchor mints @ 9 CU = 36 CU + 2 Gemini re-registration passes @ 20 CU = 40 CU
+  -> 76 CU
+- **Total: 1156 CU (~$22.13 at the ~$0.0191/CU rate implied by the 1080 CU dry-run estimate)**
+- Account model-count limit could not be read via API (403, role scope) — noted for any future
+  training on this account; if creation/training ever fails on a count-limit error, stop and
+  report, do not delete existing models.

qa/evidence/plate-sprint/outdoor-lora/camp/gemini_meta.json

@@ -0,0 +1,14 @@
+{
+  "image_ref": "asset_JacJoMavdMzU4VMGj3K13Bkp",
+  "requested_reference_images": true,
+  "reference_asset_id": "asset_MYs9LVhFtvEv7wJUfDmu1oCW",
+  "job_id": "job_pPSriYXXqnm6sB4Xet3msWNW",
+  "assets": [
+    {
+      "asset_id": "asset_zxqvrguC2iSXztMdrtsi4Zvu",
+      "path": "/Users/lume/worldos-session-notes/plate-sprint/camp/outdoor_lora/camp_outdoorlora_registered.png",
+      "bytes": 5319843
+    }
+  ],
+  "prompt": "Repaint this image in the painterly hand-painted CRPG environment-background style of the reference image: visible oil-brush strokes, muted painterly palette, warm firelit key light from the central campfire falling off into cool blue-violet shadow at the clearing's edge, dappled moonlight through canopy gaps. The blocky grey placeholder trees, crates, and rocks must be repainted as natural organic tree trunks with bark texture and foliage, rounded moss-covered boulders, and weathered wooden crates/bedrolls with visible grain -- never flat-faced geometric boxes or glassy/translucent prisms.\n\nCRITICAL STRUCTURE-LOCK (non-negotiable, checked pixel-by-pixel against the original): every wall, pillar, archway, doorway, staircase, tree, boulder, road edge, and prop must stay in EXACTLY its current position, size, and shape -- do NOT move, add, remove, resize, re-project, or recompose ANYTHING; only the paint and lighting treatment changes. Do NOT introduce any repeated, cloned, or tiled decorative motifs. Do NOT add any text, letters, numbers, labels, legends, map insets, UI elements, frames, or borders. Do NOT add any characters, people, figures, or creatures.\n\nThis is a TOP-DOWN DIMETRIC (2:1 isometric) game BACKGROUND PLATE viewed at a fixed ~30-degree downward camera pitch -- you are looking DOWN onto the scene from high above and to the side, exactly like a Pillars of Eternity II pre-rendered isometric level. Keep that EXACT high dimetric bird's-eye angle: there is NO horizon and NO sky band across the top of the frame -- the top of the frame is MORE ground/canopy seen from above. Do NOT tilt the camera to an eye-level, first-person, or landscape-horizon view."
+}

qa/evidence/plate-sprint/outdoor-lora/camp/panel_mapping.json

@@ -0,0 +1,5 @@
+{
+  "img_1.jpg": "CONTROL",
+  "img_2.jpg": "INCUMBENT_CAMP_V2",
+  "img_3.jpg": "CANDIDATE_CAMP_OUTDOORLORA"
+}

qa/evidence/plate-sprint/outdoor-lora/camp/panel_verdict.json

@@ -0,0 +1,28 @@
+{
+  "run": "camp-outdoor-lora-smoke-2026-07-10",
+  "arm": "OUTDOOR-LORA smoke test (#1481): mint anchor via schema-correct model_run (model_bfl-flux-1-dev + depth ControlNet str0.65/end0.6 + model_RsWEcQL2NWXwoyEodWVE2vWG, the new 18-image exterior-only LoRA, lorasScale 0.7, seed 42) -> Gemini instruction-edit re-registration (model_google-gemini-3-1-flash, structure-lock + dimetric-lock prompt, referenceImages=[minted anchor]) over camp_clearing_night_greybox.jpg.",
+  "panel_mapping": {"img_1.jpg": "CONTROL", "img_2.jpg": "INCUMBENT_CAMP_V2", "img_3.jpg": "CANDIDATE_CAMP_OUTDOORLORA"},
+  "scores": {
+    "img_1_CONTROL_poe2": [9, 9, 9, 9, 9],
+    "img_2_INCUMBENT_CAMP_V2": [6, 6, 7, 6, 6],
+    "img_3_CANDIDATE": [6, 7, 6, 8, 6.5]
+  },
+  "medians": {"CONTROL": 9.0, "INCUMBENT_CAMP_V2": 6.0, "CANDIDATE": 6.5},
+  "control_valid": false,
+  "control_integrity_note": "Control (poe2_control.jpg) scored median 9.0 with human figures visible and flagged by all 5 scorers -- same imperfect-control caveat as the forest_road panel and the CB-FOREST precedent; scorers judged craft only as instructed.",
+  "incumbent_cross_validation": "Incumbent (camp_clearing_night_v2.jpg, the room's actual adopted canonical_plate) scored median 6.0 blind, independently reproducing its already-registered backdrop_score=6.0 in room_recipes.json -- this cross-validates the blind-panel methodology (scorers were not told which image was which, yet landed on the known number).",
+  "candidate_defect_notes": [
+    "img_3: 6 | odd concentric-ring ground texture radiating from firepit reads as a repeated/tiled artifact, hanging bags on tree at right lack a visible attachment point",
+    "img_3: 7 | clean brushwork and lighting, faint repetitive concentric rings in the dirt around the firepit read slightly artificial",
+    "img_3: 6 | concentric ring/banding pattern in dirt around firepit looks like an artificial glow-radius artifact rather than natural ground texture",
+    "img_3: 8 | hanging satchels upper-right look faintly disconnected/floating from the branch, no characters or structures, otherwise clean",
+    "img_3: 6.5 | decent fire/smoke and moss-rock brushwork, but crate stack has a mildly ambiguous overlap/perspective and hanging satchels read slightly flat/pasted-on"
+  ],
+  "advisory_metrics": {
+    "ncc_candidate_vs_base": 0.8749,
+    "ncc_candidate_vs_greybox": 0.4558,
+    "edge_recall_vs_greybox": 0.6645,
+    "edge_recall_caveat": "Content-blind per #1491 -- lower than forest_road's number DESPITE being the visually cleaner, better result, because the candidate correctly reinterpreted the greybox's rectangular placeholder volumes as rounded organic forms (crates, boulders) rather than preserving their blocky edges literally. Confirms edge-recall alone is not a reliable quality signal here; the overlay + panel are the binding evidence."
+  },
+  "verdict": "BORDERLINE PASS, NOT a clean win. Candidate median 6.5 vs incumbent median 6.0 -- a +0.5 delta over the existing adopted plate, sitting exactly AT (not clearly above) the >6.5 success threshold, and well short of the 7.0+ target. No invented architecture, no characters, no melted/broken geometry -- the LoRA generalizes CLEANLY to camp, in sharp contrast to forest_road's severe regression. One consistent, real defect: 3/5 scorers independently flagged a repeated concentric-ring ground-texture artifact around the campfire (likely a training-data or diffusion-radial-falloff tell) and 2/5 flagged hanging-satchel attachment ambiguity. Given the borderline number and this recurring artifact, this is NOT a strong enough result to recommend replacing the adopted camp_clearing_night_v2 canonical_plate on its own -- it would need at least one more iteration (prompt-level fix for the ring artifact) before being a genuine promote.py-grade candidate."
+}

qa/evidence/plate-sprint/outdoor-lora/forest_road/gemini_meta.json

@@ -0,0 +1,14 @@
+{
+  "image_ref": "asset_rbKpGkes716PxfvTa9pmfee3",
+  "requested_reference_images": true,
+  "reference_asset_id": "asset_QwasHA86jHyJmRib1q5s3bUK",
+  "job_id": "job_mLNVWjrNLCRDfyRHwEhUUyJV",
+  "assets": [
+    {
+      "asset_id": "asset_1n3pD4HasyJ6Lwx1TyS9WYYd",
+      "path": "/Users/lume/worldos-session-notes/plate-sprint/forest_road/outdoor_lora/fr_outdoorlora_registered.png",
+      "bytes": 5753219
+    }
+  ],
+  "prompt": "Repaint this image in the painterly hand-painted CRPG environment-background style of the reference image: visible oil-brush strokes, muted painterly palette, warm-amber dusk key light raking down the dirt road falling off into cool blue-violet shadow toward the dense forest flanks. The blocky grey placeholder trees and boulders must be repainted as natural, organic tree trunks with bark texture and foliage, and rounded moss-covered boulders -- never flat-faced geometric boxes or glassy/translucent prisms.\n\nCRITICAL STRUCTURE-LOCK (non-negotiable, checked pixel-by-pixel against the original): every wall, pillar, archway, doorway, staircase, tree, boulder, road edge, and prop must stay in EXACTLY its current position, size, and shape -- do NOT move, add, remove, resize, re-project, or recompose ANYTHING; only the paint and lighting treatment changes. Do NOT introduce any repeated, cloned, or tiled decorative motifs. Do NOT add any text, letters, numbers, labels, legends, map insets, UI elements, frames, or borders. Do NOT add any characters, people, figures, or creatures.\n\nThis is a TOP-DOWN DIMETRIC (2:1 isometric) game BACKGROUND PLATE viewed at a fixed ~30-degree downward camera pitch -- you are looking DOWN onto the scene from high above and to the side, exactly like a Pillars of Eternity II pre-rendered isometric level. Keep that EXACT high dimetric bird's-eye angle: there is NO horizon and NO sky band across the top of the frame -- the top of the frame is MORE ground/canopy seen from above. Do NOT tilt the camera to an eye-level, first-person, or landscape-horizon view."
+}

qa/evidence/plate-sprint/outdoor-lora/forest_road/panel_mapping.json

@@ -0,0 +1,5 @@
+{
+  "img_1.jpg": "ANCHOR_CAMP_V2",
+  "img_2.jpg": "CONTROL",
+  "img_3.jpg": "CANDIDATE_FR_OUTDOORLORA"
+}

qa/evidence/plate-sprint/outdoor-lora/forest_road/panel_verdict.json

@@ -0,0 +1,27 @@
+{
+  "run": "forest_road-outdoor-lora-smoke-2026-07-10",
+  "arm": "OUTDOOR-LORA smoke test (#1481): mint anchor via schema-correct model_run (model_bfl-flux-1-dev + depth ControlNet str0.65/end0.6 + model_RsWEcQL2NWXwoyEodWVE2vWG, the new 18-image exterior-only LoRA, lorasScale 0.7, seed 7) -> Gemini instruction-edit re-registration (model_google-gemini-3-1-flash, structure-lock + dimetric-lock prompt, referenceImages=[minted anchor]) over forest_road_greybox.png.",
+  "panel_mapping": {"img_1.jpg": "ANCHOR_CAMP_V2", "img_2.jpg": "CONTROL", "img_3.jpg": "CANDIDATE_FR_OUTDOORLORA"},
+  "scores": {
+    "img_1_ANCHOR_CAMP_V2": [7, 7, 7, 8, 7],
+    "img_2_CONTROL_poe2": [9, 9, 9, 9, 9],
+    "img_3_CANDIDATE": [2, 2, 4, 3, 2]
+  },
+  "medians": {"ANCHOR_CAMP_V2": 7.0, "CONTROL": 9.0, "CANDIDATE": 2.0},
+  "control_valid": false,
+  "control_integrity_note": "Control (poe2_control.jpg) scored median 9.0 with human figures visible and flagged by all 5 scorers -- consistent with the CB-FOREST precedent's finding that this control is imperfect for genre-fit judgment, but scorers here were asked to judge craft only, and did.",
+  "candidate_defect_notes": [
+    "img_3: 2 | broken/nonsensical geometry -- three structures render as transparent wireframe frames with wood-carved panels floating disconnected from walls/roofs, no coherent solid form, reads as melted/AI-artifacted architecture",
+    "img_3: 2 | broken/melted geometry -- the two forest structures read as disconnected wireframe boxes with doors/curtains floating in impossible non-volumetric frames, repeated carved-door and curtain motifs tiled across separate huts",
+    "img_3: 4 | wooden hut modules read as flat disconnected box-frames with unnatural thin wireframe-like edge seams, nonsensical open-cubicle architecture with no coherent walls/roofline",
+    "img_3: 3 | nonsensical wireframe/open-frame box structures with disconnected beams and textures applied to only some faces -- reads as broken/melted geometry",
+    "img_3: 2 | broken/melted geometry -- 'buildings' are disconnected wireframe-edge fragments with walls that don't close or support each other, nonsensical structure placement in the forest, incoherent construction logic"
+  ],
+  "advisory_metrics": {
+    "ncc_candidate_vs_base": 0.6974,
+    "ncc_candidate_vs_greybox": -0.0800,
+    "edge_recall_vs_greybox": 0.9678,
+    "edge_recall_caveat": "MISLEADINGLY HIGH -- content-blind per #1491. The candidate preserved the greybox's blocky rectangular edges too literally, reading them as walls/doors instead of trees, so edges 'recall' well while content is badly wrong. The overlay image (overlay_fr_outdoorlora.png) is the binding evidence, not this number."
+  },
+  "verdict": "NOT CONVERGED -- SEVERE REGRESSION. Candidate median 2.0 is far below both the 6.0 baseline cap and the >6.5 success bar, and is WORSE than ARM C's own (interior LoRA) forest_road attempt (CB-FOREST precedent, median 6.0 tie, qa/evidence/... /cb_forest/panel_verdict.json -- kept in session-notes, not yet migrated to this evidence tree). All 5 scorers independently and unprompted flagged broken/wireframe/melted architecture -- the SAME invented-architecture failure mode the OUTDOOR-LORA effort was built to fix, still present despite training on 18 char-free, quality-passed, exterior-only images (with fortress/temple content deliberately trimmed in the pre-train quality pass). ROOT CAUSE HYPOTHESIS: forest_road's greybox represents its dense tree line as adjacent rectangular box volumes (see forest_road_greybox.png / forest_road_greybox_depth.png) -- a coarser, more literally 'architectural-looking' placeholder geometry than camp's greybox. The model appears to read blocky rectangular volumes as buildings/structures regardless of whether its training data was interior or exterior -- a GEOMETRY-SHAPE bias, not purely a training-domain bias. This is a genuinely new finding beyond the original hypothesis (that exterior-only training data would fix outdoor generalization) and suggests forest_road-class rooms may need a differently-shaped greybox (more organic/clustered tree footprints, not axis-aligned boxes) rather than a different LoRA, if this failure mode is to be fixed."
+}