diff --git a/qa/evidence/plate-sprint/outdoor-lora/SMOKE-TEST-VERDICT.md b/qa/evidence/plate-sprint/outdoor-lora/SMOKE-TEST-VERDICT.md new file mode 100644 index 00000000..5f6b7ee0 --- /dev/null +++ b/qa/evidence/plate-sprint/outdoor-lora/SMOKE-TEST-VERDICT.md @@ -0,0 +1,68 @@ +# OUTDOOR-LORA smoke test verdict (2026-07-10) + +Model: `model_RsWEcQL2NWXwoyEodWVE2vWG` ("WorldOS Painterly Exterior (FLUX)"), trained on the +18-image quality-passed set in `training_manifest.md` (job `job_U3HDsSj7T4aPETy7MRCuX6oK`, +1080 CU, ~172 min). + +Pipeline per room: mint anchor via schema-correct `model_run` (`model_bfl-flux-1-dev` + depth +ControlNet str 0.65/end 0.6 + the new LoRA, `lorasScale` 0.7) -> Gemini instruction-edit +re-registration (`model_google-gemini-3-1-flash`, structure-lock + dimetric-lock prompt, +`referenceImages=[minted anchor]`) over the room's own greybox -> overlay + advisory edge-recall +(`qa/plate_overlays.py` / `advisory_eval.py`, content-blind per #1491) -> 5-scorer blind panel +(control + reference/incumbent + candidate, shuffled, `qa/scores_db.py` surface=visual). + +## Results + +| Room | Candidate median | Reference | Baseline cap | Success bar | Result | +|---|---|---|---|---|---| +| forest_road | **2.0 / 10** | cross-lane anchor (camp_clearing_night_v2) 7.0, control 9.0 | 6.0 | >6.5 | **SEVERE REGRESSION** | +| camp_clearing_night | **6.5 / 10** | incumbent (camp_clearing_night_v2, actual adopted plate) 6.0, control 9.0 | 6.0 | >6.5 | **BORDERLINE** (right at the line, not clearly above) | + +Full scores, mapping, and defect notes: `forest_road/panel_verdict.json`, `camp/panel_verdict.json`. +scores_db rows: `outdoor-lora-smoke-forest_road-2026-07-10`, `outdoor-lora-smoke-camp-2026-07-10`. + +## Headline finding + +The new exterior-only, quality-passed LoRA **does not uniformly fix the outdoor generalization +gap** the effort was built to close (issue #1481). It generalizes cleanly to **camp** +(no invented architecture, no characters, modest +0.5 median lift over the incumbent — though +short of a clean pass) but **regresses severely on forest_road**: all 5 blind scorers +independently flagged broken/wireframe/"melted" architecture — the greybox's dense tree-line box +volumes got repainted as disconnected wooden shrine/hut structures with carved doors and +curtains, not trees. This is the SAME invented-architecture failure mode ARM C (the interior +LoRA) produced on this room (CB-FOREST precedent, median 6.0 there) — just worse (median 2.0) +and with a different visual signature (broken wireframe vs solid stone). + +**Root-cause hypothesis (new, beyond the original training-data-domain framing):** +forest_road's greybox represents its tree line as adjacent axis-aligned rectangular box volumes +(`forest_road_greybox.png` / `forest_road_greybox_depth.png`) — a coarser, more literally +"architectural-looking" placeholder shape than camp's greybox. The model appears to read blocky +rectangular volumes as buildings regardless of whether its LoRA was trained on interior or +exterior images — a **geometry-shape bias**, not purely a training-domain bias. Edge-recall +(the automated advisory metric) is misleading here: forest_road scored a deceptively HIGH 0.9678 +specifically because the candidate preserved the box edges too literally (as walls, not trees), +while camp's genuinely better result scored a LOWER 0.6645 because it correctly reinterpreted +the boxes as rounded organic forms. The overlay images are the binding evidence, not the number +(confirmed both visually and by the 5-scorer panel, independent of the metric). + +## What this means for issue #1481 + +- The OUTDOOR-LORA is a genuine, validated improvement for **camp-class** rooms (open clearings + with sparse, discrete tree/prop placeholders) but is **not ready to replace** ARM C or be + declared the fix for forest_road-class rooms (dense linear tree corridors with adjacent box + placeholders) without either (a) a differently-authored greybox for that structure class + (more organic/clustered tree footprints instead of axis-aligned boxes), or (b) further LoRA/ + prompt iteration specifically targeting box-shaped placeholder reinterpretation. +- Do not adopt either candidate as a new canonical_plate off this single smoke test. camp's + result is a borderline-promising lead worth one more iteration (fix the firepit ring artifact); + forest_road's result should be treated as a confirmed non-fix. + +## Cost actuals (record on #1481) + +- Training: 1080 CU (job_U3HDsSj7T4aPETy7MRCuX6oK, ~172 min, model_RsWEcQL2NWXwoyEodWVE2vWG) +- Smoke test: 4 anchor mints @ 9 CU = 36 CU + 2 Gemini re-registration passes @ 20 CU = 40 CU + -> 76 CU +- **Total: 1156 CU (~$22.13 at the ~$0.0191/CU rate implied by the 1080 CU dry-run estimate)** +- Account model-count limit could not be read via API (403, role scope) — noted for any future + training on this account; if creation/training ever fails on a count-limit error, stop and + report, do not delete existing models. diff --git a/qa/evidence/plate-sprint/outdoor-lora/camp/anchor_seed42.png b/qa/evidence/plate-sprint/outdoor-lora/camp/anchor_seed42.png new file mode 100644 index 00000000..35c22e2b Binary files /dev/null and b/qa/evidence/plate-sprint/outdoor-lora/camp/anchor_seed42.png differ diff --git a/qa/evidence/plate-sprint/outdoor-lora/camp/camp_outdoorlora_registered.png b/qa/evidence/plate-sprint/outdoor-lora/camp/camp_outdoorlora_registered.png new file mode 100644 index 00000000..f09c848e Binary files /dev/null and b/qa/evidence/plate-sprint/outdoor-lora/camp/camp_outdoorlora_registered.png differ diff --git a/qa/evidence/plate-sprint/outdoor-lora/camp/gemini_meta.json b/qa/evidence/plate-sprint/outdoor-lora/camp/gemini_meta.json new file mode 100644 index 00000000..9676531d --- /dev/null +++ b/qa/evidence/plate-sprint/outdoor-lora/camp/gemini_meta.json @@ -0,0 +1,14 @@ +{ + "image_ref": "asset_JacJoMavdMzU4VMGj3K13Bkp", + "requested_reference_images": true, + "reference_asset_id": "asset_MYs9LVhFtvEv7wJUfDmu1oCW", + "job_id": "job_pPSriYXXqnm6sB4Xet3msWNW", + "assets": [ + { + "asset_id": "asset_zxqvrguC2iSXztMdrtsi4Zvu", + "path": "/Users/lume/worldos-session-notes/plate-sprint/camp/outdoor_lora/camp_outdoorlora_registered.png", + "bytes": 5319843 + } + ], + "prompt": "Repaint this image in the painterly hand-painted CRPG environment-background style of the reference image: visible oil-brush strokes, muted painterly palette, warm firelit key light from the central campfire falling off into cool blue-violet shadow at the clearing's edge, dappled moonlight through canopy gaps. The blocky grey placeholder trees, crates, and rocks must be repainted as natural organic tree trunks with bark texture and foliage, rounded moss-covered boulders, and weathered wooden crates/bedrolls with visible grain -- never flat-faced geometric boxes or glassy/translucent prisms.\n\nCRITICAL STRUCTURE-LOCK (non-negotiable, checked pixel-by-pixel against the original): every wall, pillar, archway, doorway, staircase, tree, boulder, road edge, and prop must stay in EXACTLY its current position, size, and shape -- do NOT move, add, remove, resize, re-project, or recompose ANYTHING; only the paint and lighting treatment changes. Do NOT introduce any repeated, cloned, or tiled decorative motifs. Do NOT add any text, letters, numbers, labels, legends, map insets, UI elements, frames, or borders. Do NOT add any characters, people, figures, or creatures.\n\nThis is a TOP-DOWN DIMETRIC (2:1 isometric) game BACKGROUND PLATE viewed at a fixed ~30-degree downward camera pitch -- you are looking DOWN onto the scene from high above and to the side, exactly like a Pillars of Eternity II pre-rendered isometric level. Keep that EXACT high dimetric bird's-eye angle: there is NO horizon and NO sky band across the top of the frame -- the top of the frame is MORE ground/canopy seen from above. Do NOT tilt the camera to an eye-level, first-person, or landscape-horizon view." +} \ No newline at end of file diff --git a/qa/evidence/plate-sprint/outdoor-lora/camp/overlay_camp_outdoorlora.png b/qa/evidence/plate-sprint/outdoor-lora/camp/overlay_camp_outdoorlora.png new file mode 100644 index 00000000..5fab3ad6 Binary files /dev/null and b/qa/evidence/plate-sprint/outdoor-lora/camp/overlay_camp_outdoorlora.png differ diff --git a/qa/evidence/plate-sprint/outdoor-lora/camp/panel_mapping.json b/qa/evidence/plate-sprint/outdoor-lora/camp/panel_mapping.json new file mode 100644 index 00000000..4d1317b8 --- /dev/null +++ b/qa/evidence/plate-sprint/outdoor-lora/camp/panel_mapping.json @@ -0,0 +1,5 @@ +{ + "img_1.jpg": "CONTROL", + "img_2.jpg": "INCUMBENT_CAMP_V2", + "img_3.jpg": "CANDIDATE_CAMP_OUTDOORLORA" +} \ No newline at end of file diff --git a/qa/evidence/plate-sprint/outdoor-lora/camp/panel_verdict.json b/qa/evidence/plate-sprint/outdoor-lora/camp/panel_verdict.json new file mode 100644 index 00000000..e1a1c876 --- /dev/null +++ b/qa/evidence/plate-sprint/outdoor-lora/camp/panel_verdict.json @@ -0,0 +1,28 @@ +{ + "run": "camp-outdoor-lora-smoke-2026-07-10", + "arm": "OUTDOOR-LORA smoke test (#1481): mint anchor via schema-correct model_run (model_bfl-flux-1-dev + depth ControlNet str0.65/end0.6 + model_RsWEcQL2NWXwoyEodWVE2vWG, the new 18-image exterior-only LoRA, lorasScale 0.7, seed 42) -> Gemini instruction-edit re-registration (model_google-gemini-3-1-flash, structure-lock + dimetric-lock prompt, referenceImages=[minted anchor]) over camp_clearing_night_greybox.jpg.", + "panel_mapping": {"img_1.jpg": "CONTROL", "img_2.jpg": "INCUMBENT_CAMP_V2", "img_3.jpg": "CANDIDATE_CAMP_OUTDOORLORA"}, + "scores": { + "img_1_CONTROL_poe2": [9, 9, 9, 9, 9], + "img_2_INCUMBENT_CAMP_V2": [6, 6, 7, 6, 6], + "img_3_CANDIDATE": [6, 7, 6, 8, 6.5] + }, + "medians": {"CONTROL": 9.0, "INCUMBENT_CAMP_V2": 6.0, "CANDIDATE": 6.5}, + "control_valid": false, + "control_integrity_note": "Control (poe2_control.jpg) scored median 9.0 with human figures visible and flagged by all 5 scorers -- same imperfect-control caveat as the forest_road panel and the CB-FOREST precedent; scorers judged craft only as instructed.", + "incumbent_cross_validation": "Incumbent (camp_clearing_night_v2.jpg, the room's actual adopted canonical_plate) scored median 6.0 blind, independently reproducing its already-registered backdrop_score=6.0 in room_recipes.json -- this cross-validates the blind-panel methodology (scorers were not told which image was which, yet landed on the known number).", + "candidate_defect_notes": [ + "img_3: 6 | odd concentric-ring ground texture radiating from firepit reads as a repeated/tiled artifact, hanging bags on tree at right lack a visible attachment point", + "img_3: 7 | clean brushwork and lighting, faint repetitive concentric rings in the dirt around the firepit read slightly artificial", + "img_3: 6 | concentric ring/banding pattern in dirt around firepit looks like an artificial glow-radius artifact rather than natural ground texture", + "img_3: 8 | hanging satchels upper-right look faintly disconnected/floating from the branch, no characters or structures, otherwise clean", + "img_3: 6.5 | decent fire/smoke and moss-rock brushwork, but crate stack has a mildly ambiguous overlap/perspective and hanging satchels read slightly flat/pasted-on" + ], + "advisory_metrics": { + "ncc_candidate_vs_base": 0.8749, + "ncc_candidate_vs_greybox": 0.4558, + "edge_recall_vs_greybox": 0.6645, + "edge_recall_caveat": "Content-blind per #1491 -- lower than forest_road's number DESPITE being the visually cleaner, better result, because the candidate correctly reinterpreted the greybox's rectangular placeholder volumes as rounded organic forms (crates, boulders) rather than preserving their blocky edges literally. Confirms edge-recall alone is not a reliable quality signal here; the overlay + panel are the binding evidence." + }, + "verdict": "BORDERLINE PASS, NOT a clean win. Candidate median 6.5 vs incumbent median 6.0 -- a +0.5 delta over the existing adopted plate, sitting exactly AT (not clearly above) the >6.5 success threshold, and well short of the 7.0+ target. No invented architecture, no characters, no melted/broken geometry -- the LoRA generalizes CLEANLY to camp, in sharp contrast to forest_road's severe regression. One consistent, real defect: 3/5 scorers independently flagged a repeated concentric-ring ground-texture artifact around the campfire (likely a training-data or diffusion-radial-falloff tell) and 2/5 flagged hanging-satchel attachment ambiguity. Given the borderline number and this recurring artifact, this is NOT a strong enough result to recommend replacing the adopted camp_clearing_night_v2 canonical_plate on its own -- it would need at least one more iteration (prompt-level fix for the ring artifact) before being a genuine promote.py-grade candidate." +} diff --git a/qa/evidence/plate-sprint/outdoor-lora/forest_road/anchor_seed7.png b/qa/evidence/plate-sprint/outdoor-lora/forest_road/anchor_seed7.png new file mode 100644 index 00000000..12252329 Binary files /dev/null and b/qa/evidence/plate-sprint/outdoor-lora/forest_road/anchor_seed7.png differ diff --git a/qa/evidence/plate-sprint/outdoor-lora/forest_road/fr_outdoorlora_registered.png b/qa/evidence/plate-sprint/outdoor-lora/forest_road/fr_outdoorlora_registered.png new file mode 100644 index 00000000..f1f86441 Binary files /dev/null and b/qa/evidence/plate-sprint/outdoor-lora/forest_road/fr_outdoorlora_registered.png differ diff --git a/qa/evidence/plate-sprint/outdoor-lora/forest_road/gemini_meta.json b/qa/evidence/plate-sprint/outdoor-lora/forest_road/gemini_meta.json new file mode 100644 index 00000000..f5734bdf --- /dev/null +++ b/qa/evidence/plate-sprint/outdoor-lora/forest_road/gemini_meta.json @@ -0,0 +1,14 @@ +{ + "image_ref": "asset_rbKpGkes716PxfvTa9pmfee3", + "requested_reference_images": true, + "reference_asset_id": "asset_QwasHA86jHyJmRib1q5s3bUK", + "job_id": "job_mLNVWjrNLCRDfyRHwEhUUyJV", + "assets": [ + { + "asset_id": "asset_1n3pD4HasyJ6Lwx1TyS9WYYd", + "path": "/Users/lume/worldos-session-notes/plate-sprint/forest_road/outdoor_lora/fr_outdoorlora_registered.png", + "bytes": 5753219 + } + ], + "prompt": "Repaint this image in the painterly hand-painted CRPG environment-background style of the reference image: visible oil-brush strokes, muted painterly palette, warm-amber dusk key light raking down the dirt road falling off into cool blue-violet shadow toward the dense forest flanks. The blocky grey placeholder trees and boulders must be repainted as natural, organic tree trunks with bark texture and foliage, and rounded moss-covered boulders -- never flat-faced geometric boxes or glassy/translucent prisms.\n\nCRITICAL STRUCTURE-LOCK (non-negotiable, checked pixel-by-pixel against the original): every wall, pillar, archway, doorway, staircase, tree, boulder, road edge, and prop must stay in EXACTLY its current position, size, and shape -- do NOT move, add, remove, resize, re-project, or recompose ANYTHING; only the paint and lighting treatment changes. Do NOT introduce any repeated, cloned, or tiled decorative motifs. Do NOT add any text, letters, numbers, labels, legends, map insets, UI elements, frames, or borders. Do NOT add any characters, people, figures, or creatures.\n\nThis is a TOP-DOWN DIMETRIC (2:1 isometric) game BACKGROUND PLATE viewed at a fixed ~30-degree downward camera pitch -- you are looking DOWN onto the scene from high above and to the side, exactly like a Pillars of Eternity II pre-rendered isometric level. Keep that EXACT high dimetric bird's-eye angle: there is NO horizon and NO sky band across the top of the frame -- the top of the frame is MORE ground/canopy seen from above. Do NOT tilt the camera to an eye-level, first-person, or landscape-horizon view." +} \ No newline at end of file diff --git a/qa/evidence/plate-sprint/outdoor-lora/forest_road/overlay_fr_outdoorlora.png b/qa/evidence/plate-sprint/outdoor-lora/forest_road/overlay_fr_outdoorlora.png new file mode 100644 index 00000000..84384675 Binary files /dev/null and b/qa/evidence/plate-sprint/outdoor-lora/forest_road/overlay_fr_outdoorlora.png differ diff --git a/qa/evidence/plate-sprint/outdoor-lora/forest_road/panel_mapping.json b/qa/evidence/plate-sprint/outdoor-lora/forest_road/panel_mapping.json new file mode 100644 index 00000000..11057922 --- /dev/null +++ b/qa/evidence/plate-sprint/outdoor-lora/forest_road/panel_mapping.json @@ -0,0 +1,5 @@ +{ + "img_1.jpg": "ANCHOR_CAMP_V2", + "img_2.jpg": "CONTROL", + "img_3.jpg": "CANDIDATE_FR_OUTDOORLORA" +} \ No newline at end of file diff --git a/qa/evidence/plate-sprint/outdoor-lora/forest_road/panel_verdict.json b/qa/evidence/plate-sprint/outdoor-lora/forest_road/panel_verdict.json new file mode 100644 index 00000000..b18bd37a --- /dev/null +++ b/qa/evidence/plate-sprint/outdoor-lora/forest_road/panel_verdict.json @@ -0,0 +1,27 @@ +{ + "run": "forest_road-outdoor-lora-smoke-2026-07-10", + "arm": "OUTDOOR-LORA smoke test (#1481): mint anchor via schema-correct model_run (model_bfl-flux-1-dev + depth ControlNet str0.65/end0.6 + model_RsWEcQL2NWXwoyEodWVE2vWG, the new 18-image exterior-only LoRA, lorasScale 0.7, seed 7) -> Gemini instruction-edit re-registration (model_google-gemini-3-1-flash, structure-lock + dimetric-lock prompt, referenceImages=[minted anchor]) over forest_road_greybox.png.", + "panel_mapping": {"img_1.jpg": "ANCHOR_CAMP_V2", "img_2.jpg": "CONTROL", "img_3.jpg": "CANDIDATE_FR_OUTDOORLORA"}, + "scores": { + "img_1_ANCHOR_CAMP_V2": [7, 7, 7, 8, 7], + "img_2_CONTROL_poe2": [9, 9, 9, 9, 9], + "img_3_CANDIDATE": [2, 2, 4, 3, 2] + }, + "medians": {"ANCHOR_CAMP_V2": 7.0, "CONTROL": 9.0, "CANDIDATE": 2.0}, + "control_valid": false, + "control_integrity_note": "Control (poe2_control.jpg) scored median 9.0 with human figures visible and flagged by all 5 scorers -- consistent with the CB-FOREST precedent's finding that this control is imperfect for genre-fit judgment, but scorers here were asked to judge craft only, and did.", + "candidate_defect_notes": [ + "img_3: 2 | broken/nonsensical geometry -- three structures render as transparent wireframe frames with wood-carved panels floating disconnected from walls/roofs, no coherent solid form, reads as melted/AI-artifacted architecture", + "img_3: 2 | broken/melted geometry -- the two forest structures read as disconnected wireframe boxes with doors/curtains floating in impossible non-volumetric frames, repeated carved-door and curtain motifs tiled across separate huts", + "img_3: 4 | wooden hut modules read as flat disconnected box-frames with unnatural thin wireframe-like edge seams, nonsensical open-cubicle architecture with no coherent walls/roofline", + "img_3: 3 | nonsensical wireframe/open-frame box structures with disconnected beams and textures applied to only some faces -- reads as broken/melted geometry", + "img_3: 2 | broken/melted geometry -- 'buildings' are disconnected wireframe-edge fragments with walls that don't close or support each other, nonsensical structure placement in the forest, incoherent construction logic" + ], + "advisory_metrics": { + "ncc_candidate_vs_base": 0.6974, + "ncc_candidate_vs_greybox": -0.0800, + "edge_recall_vs_greybox": 0.9678, + "edge_recall_caveat": "MISLEADINGLY HIGH -- content-blind per #1491. The candidate preserved the greybox's blocky rectangular edges too literally, reading them as walls/doors instead of trees, so edges 'recall' well while content is badly wrong. The overlay image (overlay_fr_outdoorlora.png) is the binding evidence, not this number." + }, + "verdict": "NOT CONVERGED -- SEVERE REGRESSION. Candidate median 2.0 is far below both the 6.0 baseline cap and the >6.5 success bar, and is WORSE than ARM C's own (interior LoRA) forest_road attempt (CB-FOREST precedent, median 6.0 tie, qa/evidence/... /cb_forest/panel_verdict.json -- kept in session-notes, not yet migrated to this evidence tree). All 5 scorers independently and unprompted flagged broken/wireframe/melted architecture -- the SAME invented-architecture failure mode the OUTDOOR-LORA effort was built to fix, still present despite training on 18 char-free, quality-passed, exterior-only images (with fortress/temple content deliberately trimmed in the pre-train quality pass). ROOT CAUSE HYPOTHESIS: forest_road's greybox represents its dense tree line as adjacent rectangular box volumes (see forest_road_greybox.png / forest_road_greybox_depth.png) -- a coarser, more literally 'architectural-looking' placeholder geometry than camp's greybox. The model appears to read blocky rectangular volumes as buildings/structures regardless of whether its training data was interior or exterior -- a GEOMETRY-SHAPE bias, not purely a training-domain bias. This is a genuinely new finding beyond the original hypothesis (that exterior-only training data would fix outdoor generalization) and suggests forest_road-class rooms may need a differently-shaped greybox (more organic/clustered tree footprints, not axis-aligned boxes) rather than a different LoRA, if this failure mode is to be fixed." +} diff --git a/qa/evidence/plate-sprint/outdoor-lora/model_state.json b/qa/evidence/plate-sprint/outdoor-lora/model_state.json new file mode 100644 index 00000000..aeb7be43 --- /dev/null +++ b/qa/evidence/plate-sprint/outdoor-lora/model_state.json @@ -0,0 +1,67 @@ +{ + "model_id": "model_RsWEcQL2NWXwoyEodWVE2vWG", + "model_name": "WorldOS Painterly Exterior (FLUX)", + "model_type": "flux.1-lora", + "scenario_project_id": "proj_sd4w3ozsJaBHYnHS4mP1S2Pw", + "status": "training", + "training_job_id": "job_U3HDsSj7T4aPETy7MRCuX6oK", + "training_started_at": "2026-07-10T10:21:49.874Z", + "training_config": { + "rank": 64, + "learningRate": 0.0001, + "learningRateTextEncoder": 0.00001, + "nbEpochs": 10, + "nbRepeats": 20, + "batchSize": 1, + "note": "mirrors ARM C's (model_G379oza2qhm6MkqDrtTvvmmw) proven config for apples-to-apples smoke-test comparability" + }, + "blocker_resolved": { + "type": "PlanLimitReachedError", + "http_status": 429, + "actionName": "train-model", + "actionLimit": 10000, + "actionValue_at_block": 9754, + "limitScope": "team", + "resolution": "Owner added 5,000 CU to the Scenario account on 2026-07-10 after confirming (7 private models on the account, none an exterior FLUX LoRA) this training is not redundant. train(action=start) succeeded after this." + }, + "quality_pass_before_training": { + "performed": "2026-07-10, immediately before train(action=start)", + "char_free_zoom_recheck": "all 18 confirmed char-free at full resolution (not thumbnail); one ambiguous figure (gilded_vale_wilderness) crop-zoomed 3x and confirmed a statue on a pedestal, not a living figure", + "resolution_compression_watermark_check": "all clean, no watermarks, no compression artifacts; two resolution tiers (2048 vs 2560-2758px long edge -- 5 reused v4 assets vs 13 this-session uploads) noted as minor, non-blocking", + "caption_vs_content_mismatches_found": "t_046 (labeled temple_abbey) and t_048 (labeled ruins_firstfires) actually depict grander architecture (an ice fortress; a full cathedral plaza) than their PoE-wiki filenames implied; ext_017 (bog) was Scenario auto-caption-misfired as a 'war-torn landscape...tanks' scene -- visually confirmed a legitimate marsh/ruins scene, no vehicles, no characters", + "rebalance_action": "swapped OUT t_048/ruins_firstfires (asset_dWCdqiENwCyq9EdzDDQCfUTK, a full cathedral-plaza -- the single most architecture-dominant frame, and least representative of the forest_road/camp smoke-test target rooms) for refc_068/western_exterior_mountain (asset_XhU3yw8GWrJx5gWLFxsrPimf, natural snowy terrain, zero architecture, already uploaded from the earlier cost-trim pass at no extra cost)", + "dedupe_check": "no near-identical frames in the final 18 (the one true duplicate scene in the wider corpus, refc_066 vs t_044, was already resolved pre-quality-pass by using only t_044)", + "cost_after_swap": "dry-run reconfirmed 1080 CU (unchanged -- image count is the cost driver, not composition)" + }, + "dry_run_cost_cu": 1080, + "dry_run_cost_estimate_usd": 20.67, + "lane_cap_usd": 25, + "account_model_count_limit_note": "Could not be read via API (403, role scope) -- if a future model create/train call ever fails on a count-limit error: STOP and report, do not delete existing models to work around it.", + "training_images_final_18": [ + {"asset_id": "asset_58ehPSyWAB4dVpWmp4oNg9LS", "file": "ext_001_fortress_raedric_hold.jpg", "reused_from": "worldos-poe2-clean-env-v4 (t_044.jpg)"}, + {"asset_id": "asset_fdVzG3S1Y4jyR8ErCLqLCyvq", "file": "ext_002_ruins_oldsong.jpg", "reused_from": "worldos-poe2-clean-env-v4 (t_045.jpg)"}, + {"asset_id": "asset_pkgacsDZsqxXeMRrp8bNZVRK", "file": "ext_003_temple_abbey.jpg", "reused_from": "worldos-poe2-clean-env-v4 (t_046.jpg)"}, + {"asset_id": "asset_uzqFWgHgyghpnmM3QFDyfvKi", "file": "ext_004_settlement_brackenbury.jpg", "reused_from": "worldos-poe2-clean-env-v4 (t_047.jpg)"}, + {"asset_id": "asset_XhU3yw8GWrJx5gWLFxsrPimf", "file": "ext_005_western_exterior_mountain.jpg", "new_upload": true, "note": "swapped in during quality pass, replacing ruins_firstfires"}, + {"asset_id": "asset_KwmxyNGS274jtUWMZ3BbpfHC", "file": "ext_006_jungle_path.jpg", "new_upload": true}, + {"asset_id": "asset_hWtoxJFBN1i6UeJfe3CR7Gz8", "file": "ext_007_jungle_courtyard_shrine.jpg", "new_upload": true}, + {"asset_id": "asset_nYiu6Gu6GXWdRKSh9TKfMhYs", "file": "ext_008_twinelms_hearthsong_village.jpg", "new_upload": true}, + {"asset_id": "asset_ofmGDEpi6ur85RnTBgA6LAzJ", "file": "ext_009_ondras_gift_coast.jpg", "new_upload": true}, + {"asset_id": "asset_v7ADRUjCQACLPJdXh9nbsBKR", "file": "ext_010_night_camp.jpg", "new_upload": true}, + {"asset_id": "asset_SquJXAUqH9D9R8zaRXu2dztd", "file": "ext_011_gilded_vale_wilderness.jpg", "new_upload": true}, + {"asset_id": "asset_VzYgaAA38o6QuHFS9NMmqYGg", "file": "ext_012_black_meadow_wilderness.jpg", "new_upload": true}, + {"asset_id": "asset_LydPVC6jauLTGaYsmKMhPhTD", "file": "ext_013_stronghold_ext_pristine.jpg", "new_upload": true}, + {"asset_id": "asset_oR9zJaAB79sPJv7KG6N7GfUr", "file": "ext_014_bridge_district.jpg", "new_upload": true}, + {"asset_id": "asset_q26TbJau96Z2qgF2Cuw5qPzN", "file": "ext_015_madhmr_bridge.jpg", "new_upload": true}, + {"asset_id": "asset_2jzyZEd1ukzp5emeiukC1zae", "file": "ext_016_woodend_plains.jpg", "new_upload": true}, + {"asset_id": "asset_GbPoqFErs6rfifNa6QFTr18T", "file": "ext_017_bog.jpg", "new_upload": true}, + {"asset_id": "asset_kpVSMEiqkxhgtmvuMpXD8v6t", "file": "ext_018_camp_clearing_night_v2_ADOPTED.jpg", "reused_from": "ARM C training set + qa/evidence/plate-audit/camp_clearing_night_v2.jpg"} + ], + "deleted_final": [ + {"asset_id": "asset_dWCdqiENwCyq9EdzDDQCfUTK", "file": "ruins_firstfires (t_048)", "reason": "quality-pass rebalance -- most architecture-dominant frame (full cathedral plaza), swapped for western_exterior_mountain"}, + {"asset_id": "asset_3qcdgkVmFKrpzmj7StJRTT77", "file": "eastern_exterior_mountain", "reason": "cost trim + duplicate-of-western, stayed excluded"}, + {"asset_id": "asset_rqeAXPXtPJrH6sMAQCZ3Pg4S", "file": "jungle_waterfall", "reason": "cost trim; quality pass separately confirmed this has characters ('three adventurers') -- correctly excluded"}, + {"asset_id": "asset_zGGeb6YKonabKqquXGTRi128", "file": "durgans_battery_ext", "reason": "cost trim, 3rd redundant fortress-ext frame, stayed excluded"} + ], + "next_steps": "Poll job_U3HDsSj7T4aPETy7MRCuX6oK via jobs_wait/job_get to completion (ETA ~1-2h based on ARM C's comparable 111min/627CU run). Then run the smoke test: forest_road + camp greybox anchors (flux.1-dev + depth-CN + this LoRA, str~0.6-0.7) -> Gemini re-registration -> overlay + advisory edge-recall -> 5-scorer blind panel vs the 6.0 baseline. Success = outdoor median >6.5, target 7.0+." +} diff --git a/qa/evidence/plate-sprint/outdoor-lora/training_manifest.md b/qa/evidence/plate-sprint/outdoor-lora/training_manifest.md new file mode 100644 index 00000000..03855a86 --- /dev/null +++ b/qa/evidence/plate-sprint/outdoor-lora/training_manifest.md @@ -0,0 +1,174 @@ +# OUTDOOR-LORA training-set manifest + +Curated exterior-heavy training set for **`WorldOS Painterly Exterior (FLUX)`** +(`model_RsWEcQL2NWXwoyEodWVE2vWG`, Scenario, `flux.1-lora`, project +`proj_sd4w3ozsJaBHYnHS4mP1S2Pw`), assembled to counter the interior-heavy ARM C +LoRA (`model_G379oza2qhm6MkqDrtTvvmmw`, issue #1481) that regresses the +outdoor `forest_road` plate class to the 6.0 quality cap (PR #1495, +`qa/evidence/plate-sprint/cb-forest/`). + +18 images, all char-free (no human/creature figures), all real painterly +references except one adopted WorldOS-generated plate (flagged below). Sourced +from the same corpus that built the interior-heavy +`worldos-poe2-clean-env-v4` model (`model_ng6MSaWtyvwKbZyDBMtzv7tA`) — that +corpus (`~/worldos-session-notes/poe2-refs-clean/`, `manifest.json`, 77 +candidate PoE1/PoE2/Tyranny reference frames) is 43/48 interior-biased in its +curated `SELECTED.txt`; this set instead pulls the corpus's EXTERIOR +candidates that were left uncurated for v4, plus reuses v4's 5 already-curated +exteriors (Scenario asset IDs below reused directly, no re-upload). + +## Selection rule +REAL painterly references first (Fandom PoE1/PoE2 wiki "File:" pages + +RPGFan press/promo screenshots — the same source class already used and +owner-approved for `worldos-poe2-clean-env-v4`), covering forest, camp, town +street/square, coast, ruins-outdoor, and battlefield-adjacent open terrain. +Excluded: `refc_015.jpg` (Tyranny battlefield exterior) — the only frame in +the source corpus explicitly tagged as depicting a battlefield, but flagged in +`manifest.json` as having small distant NPCs visible ("may not meet strict +character-free bar") — dropped for the char-free requirement. No other clean +char-free battlefield-specific frame was found in the corpus; `bog.jpg` +(#18 below) is the closest visual stand-in (its own Scenario auto-caption +independently read it as a "war-torn landscape ... abandoned military +vehicles, rusted tank remnants" — an unplanned but useful battlefield-adjacent +texture note for a wetland/waste frame). + +Excluded from the initial 22-image pull to keep training cost inside the +lane's Scenario CU budget (see COST NOTE below): 2 duplicate-concept +"exterior mountain" frames (`refc_068`/`refc_069`, western/eastern exterior) +and 2 redundant frames (`refc_013` jungle waterfall — 3rd jungle-biome shot; +`refc_067` Durgan's Battery ext — 3rd fortress-exterior shot). Category +coverage held up without them. + +## Manifest + +| # | file (local) | source | biome | license / provenance note | Scenario asset_id | +|---|---|---|---|---|---| +| 1 | ext_001_fortress_raedric_hold.jpg | Fandom PoE1 wiki, `Ar_0707_raedrics_hold_ext.jpg` (reused from v4 curated set, `t_044.jpg`) | fortress_stronghold, exterior | Fandom community wiki, editorial/reference use, same class as owner-approved v4 corpus | `asset_58ehPSyWAB4dVpWmp4oNg9LS` (reused, no re-upload) | +| 2 | ext_002_ruins_oldsong.jpg | Fandom PoE1 wiki, `Ar_1201_oldsong_ext.jpg` (reused, `t_045.jpg`) | ruins, exterior | Fandom wiki | `asset_fdVzG3S1Y4jyR8ErCLqLCyvq` (reused) | +| 3 | ext_003_temple_abbey.jpg | Fandom PoE1 wiki, `Px2_0301_abbey_ext.jpg` (reused, `t_046.jpg`) | temple_sacred, exterior | Fandom wiki | `asset_pkgacsDZsqxXeMRrp8bNZVRK` (reused) | +| 4 | ext_004_settlement_brackenbury.jpg | Fandom PoE1 wiki, `Ar_0401_brackenbury_exterior.jpg` (reused, `t_047.jpg`) | settlement, exterior (town street) | Fandom wiki | `asset_uzqFWgHgyghpnmM3QFDyfvKi` (reused) | +| 5 | ext_005_western_exterior_mountain.jpg | Fandom wiki, `Px1_0301_western_exterior.jpg` (`refc_068.jpg`) | mountain, exterior | Fandom wiki, PoE1 | `asset_XhU3yw8GWrJx5gWLFxsrPimf` (already-uploaded, re-attached at no extra cost) | +| 6 | ext_006_jungle_path.jpg | RPGFan, `Pillars-of-Eternity-II-Deadfire-Artwork-007.jpg` (`refc_002.jpg`) | exterior (forest-road analog) | RPGFan press/promo screenshot, editorial use | `asset_KwmxyNGS274jtUWMZ3BbpfHC` | +| 7 | ext_007_jungle_courtyard_shrine.jpg | RPGFan, `...Screenshot-066.jpg` (`refc_003.jpg`) | exterior (ruins-outdoor analog) | RPGFan press screenshot | `asset_hWtoxJFBN1i6UeJfe3CR7Gz8` | +| 8 | ext_008_twinelms_hearthsong_village.jpg | Fandom wiki, `Twin-elms-hearthsong.jpg` (`refc_004.webp`) | exterior (town street) | Fandom wiki, PoE1 | `asset_nYiu6Gu6GXWdRKSh9TKfMhYs` | +| 9 | ext_009_ondras_gift_coast.jpg | Fandom wiki, `Ar_0103_def_bay_ondras_gift.jpg` (`refc_006.webp`) | settlement, exterior (coast) | Fandom wiki, PoE1 | `asset_ofmGDEpi6ur85RnTBgA6LAzJ` | +| 10 | ext_010_night_camp.jpg | RPGFan, `Pillars-of-Eternity-Screenshot-001.jpg` (`refc_007.jpg`) | exterior (camp) | RPGFan press screenshot — **URL re-derived from gallery index position, not independently re-verified this session** (flagged in source `manifest.json`) | `asset_v7ADRUjCQACLPJdXh9nbsBKR` | +| 11 | ext_011_gilded_vale_wilderness.jpg | Fandom wiki, `Ar_0701_gilded_vale_wilderness.jpg` (`refc_063.jpg`) | forest, exterior | Fandom wiki, PoE1 | `asset_SquJXAUqH9D9R8zaRXu2dztd` | +| 12 | ext_012_black_meadow_wilderness.jpg | Fandom wiki, `Ar_0801_black_meadow_wilderness.jpg` (`refc_064.jpg`) | forest, exterior | Fandom wiki, PoE1 | `asset_VzYgaAA38o6QuHFS9NMmqYGg` | +| 13 | ext_013_stronghold_ext_pristine.jpg | Fandom wiki, `Ar_0601_strongholdexterior_pristine.jpg` (`refc_065.jpg`) | fortress_stronghold, exterior | Fandom wiki, PoE1 | `asset_LydPVC6jauLTGaYsmKMhPhTD` | +| 14 | ext_014_bridge_district.jpg | Fandom wiki, `Ar_0112_bridge_district.jpg` (`refc_072.jpg`) | settlement, exterior (town street/bridge) | Fandom wiki, PoE1 | `asset_oR9zJaAB79sPJv7KG6N7GfUr` | +| 15 | ext_015_madhmr_bridge.jpg | Fandom wiki, `Ar_0718_madhmr_bridge.jpg` (`refc_073.jpg`) | settlement, exterior (town street/bridge) | Fandom wiki, PoE1 | `asset_q26TbJau96Z2qgF2Cuw5qPzN` | +| 16 | ext_016_woodend_plains.jpg | Fandom wiki, `Ar_0811_woodend_plains.jpg` (`refc_076.jpg`) | wetland_waste, exterior (open-field/battlefield-adjacent) | Fandom wiki, PoE1 | `asset_2jzyZEd1ukzp5emeiukC1zae` | +| 17 | ext_017_bog.jpg | Fandom wiki, `Px2_0502_bog.jpg` (`refc_077.jpg`) | wetland_waste, exterior (battlefield-adjacent — see selection-rule note) | Fandom wiki, PoE1 | `asset_GbPoqFErs6rfifNa6QFTr18T` | +| 18 | ext_018_camp_clearing_night_v2_ADOPTED.jpg | **WorldOS-generated, adopted** (`qa/evidence/plate-audit/camp_clearing_night_v2.jpg`, 2026-07-09) | camp, exterior | Our own adopted outdoor plate — the cross-lane "house-best" incumbent reused as a panel anchor in PR #1495/#1490; NOT a low-scoring generated slop frame (it is the adopted 6.0-9.0-range incumbent). Already independently reused as an ARM C training image (`asset_kpVSMEiqkxhgtmvuMpXD8v6t`), so this precedent existed before this lane. | `asset_kpVSMEiqkxhgtmvuMpXD8v6t` (reused, same asset as in ARM C's training set) | + +**Copyright note (transparency, not a new decision):** items 6-17 are sourced +from Fandom-wiki "File:" pages and RPGFan press/promo screenshots of Obsidian +Entertainment's Pillars of Eternity — the identical source class (same wiki, +same screenshot galleries) already used and owner-approved to build +`worldos-poe2-clean-env-v4`. This manifest documents source URL + license +class per image (the owner's stated repeatability condition) but does not +re-litigate that precedent. + +## Excluded (deliberately, this pass) +- `refc_015.jpg` — battlefield exterior, characters visible (char-free bar). +- `refc_069.jpg` — eastern "exterior mountain", duplicative of the western + frame (#5) once that was swapped back in; not a named mandate category. +- `refc_013.jpg` (jungle waterfall) — **has characters** ("three adventurers + stand on a sandy beach"), caught on the quality-pass caption re-check; + correctly excluded already for cost, now double-confirmed exclude for + char-free. +- `refc_067.jpg` (Durgan's Battery ext) — 3rd redundant fortress-exterior + frame; dropped to keep architecture-heavy content from dominating (see + Quality pass below). +- `t_048.jpg` / `refc_070.jpg` ("ruins_firstfires") — **swapped OUT** in the + quality pass (see below): visually a grand cathedral/basilica plaza + (domes, colonnades, twin statues), not simple ruins — the single most + architecture-dominant frame in the set. Swapped for `refc_068.jpg` + (western exterior mountain, already-uploaded, no extra cost) to shore up + genuine natural-terrain representation for the forest_road/camp target + rooms. + +## Quality pass (owner-directed, before training — 2026-07-10) + +Per owner discipline ("models are expensive — make prep top-notch before +spending"), every one of the 18 images was re-verified at full resolution +(not thumbnail) before `train(action=start)`, plus two were crop-zoomed for a +disputed detail: + +1. **Char-free re-check (zoom, not thumbnail):** all 18 confirmed free of + living human/creature figures. + - `ext_005` (then "gilded_vale_wilderness", now the mountain swap target) + had one ambiguous humanoid silhouette — crop-zoomed 3x + (`/tmp/gilded_vale_zoom.png`): confirmed a stone/bronze **statue on a + pedestal** (matches the same wayside-monument convention as the + confirmed statues in `ext_004` and the swapped-out `t_048`), not a + living figure. Kept. + - `ext_010` ("night_camp") — caption mentions "cave walls"; visual check + confirms it's a rock-alcove/grotto clearing, technically exterior (sky + and tree canopy visible top-left) but more enclosed than an open forest + camp. Kept — it's the corpus's only real "camp" reference — but flagged: + lean on `ext_018` (our adopted `camp_clearing_night_v2`, open-sky) as the + cleaner camp exemplar in this set. + - `ext_015` (madhmr_bridge) has 2-3 small reddish shapes in the grass — + confirmed wildlife (resting deer/animal), not humanoid figures. Common + PoE scene-dressing convention, not disqualifying. + - `refc_013.jpg` (jungle waterfall, already excluded for cost) was + independently caught here too: its caption reads "three adventurers + stand on a sandy beach" — genuinely has characters. Good thing it was + already out; this closes the loop on why. +2. **Resolution / compression / watermark check:** all 18 are clean JPEGs, + long-edge ~2048-2560px (5 reused v4 assets sit at 2560x2015-2560x2758 from + prior curation, the 13 new uploads were resized to 2048 long-edge this + session) — no corruption, no watermarks, no visible compression artifacts + at 1:1. The two resolution tiers are a minor inconsistency but not a + blocker (Scenario's training pipeline buckets/resizes inputs internally). +3. **Auto-caption vs. actual-content mismatches found** (Scenario's + auto-captioner, not a manifest error, but worth recording since it changed + my read of category balance): `t_046` ("temple_abbey") is actually + captioned/depicts a grand ice-fortress complex, not a simple abbey; + `t_048` ("ruins_firstfires") is captioned/depicts a full cathedral plaza, + not ruins; `ext_017` ("bog") was mis-captioned by Scenario as a "war-torn + landscape... abandoned military vehicles, rusted tank remnants" — visual + check confirms this is a captioning misfire, the actual image is a + legitimate marsh/ruins scene with wooden boardwalks and a skull-motif + stone entrance, no military hardware, no characters. +4. **Category-balance rebalance (the actual finding that changed the set):** + the corpus's own PoE-wiki "biome" tags undersold how architecture-heavy + several frames actually are once viewed (see point 3) — grand + fortress/castle/cathedral content was a de-facto plurality (~7-8/18) + against genuinely natural/rustic content directly analogous to the two + smoke-test target rooms, forest_road and camp (~5-6/18: jungle_path, + gilded_vale hillside-path, black_meadow tents+campfire+skeleton, + woodend_plains clearing, madhmr_bridge, camp_clearing_night_v2). This is + the same failure shape as ARM C (architecture bias invading natural + scenes), just latent rather than 10/10 dominant. **Action taken:** dropped + the single most architecture-dominant frame (`t_048`/"ruins_firstfires" — + full cathedral plaza) and swapped in `refc_068.jpg` (western exterior + mountain: snowy rocks, sparse trees, frozen ponds, zero architecture, + already uploaded from the cost-trim pass, asset_XhU3yw8GWrJx5gWLFxsrPimf) + — same 18-image count, same 1080 CU cost (re-confirmed by dry-run after + the swap), better-aimed at the target rooms. +5. **Dedupe check:** no near-identical frames found among the final 18 (the + one true duplicate scene in the wider corpus, `refc_066`/`Raedric's Hold + ext` vs `t_044`, was already resolved by using only `t_044`). + +## Cost + quota — RESOLVED, training started + +Owner added 5,000 CU to the Scenario account (2026-07-10) after model-count +inventory was checked (7 private models on the account, none an exterior +FLUX LoRA — this training is not redundant). Post-swap dry-run reconfirmed +**1080 CU** (~$20.7 est., unchanged from pre-swap — image count is the cost +driver here, not composition), still comfortably under the lane's $25 cap. + +`train(action=start)` succeeded: **job `job_U3HDsSj7T4aPETy7MRCuX6oK`**, +`flux-model-training`, `cuCost: 1080`, status `queued` → `running-train` as +of 2026-07-10T10:21 UTC. Config: `rank=64, learningRate=0.0001, +learningRateTextEncoder=0.00001, nbEpochs=10, nbRepeats=20` (mirrors ARM C's +proven config). ETA comparable to ARM C's 111 min run. + +**Note for future training on this account:** the Scenario account's +model-count limit could not be read via the API (`GET` on the relevant +endpoint returns 403, role scope). If a future model create/train call ever +fails on a count-limit error, **stop and report — do not delete existing +models** to work around it. diff --git a/qa/scores.db b/qa/scores.db index e4b65fe1..d8396227 100644 Binary files a/qa/scores.db and b/qa/scores.db differ diff --git a/qa/scores_ledger.md b/qa/scores_ledger.md index 1c020457..6445014a 100644 --- a/qa/scores_ledger.md +++ b/qa/scores_ledger.md @@ -6,10 +6,12 @@ > **Ruler** = `scoring_config_version` (a content hash of the rubric + schema + gate files, RRI gate included). **Lens ruler** = `lens_config_version` (the 8 files that produce the story/mech/angry LENS numbers — full ruler minus `release_readiness.py`; blank = recorded before lens stamping, #725). Rows under DIFFERENT Ruler values are **NOT directly comparable as a quality trend** — the ruler changed (a rubric recalibration or a new gate moves the number with no change in play quality). Use `python3 qa/scores_db.py --compare` for a lens-fenced engine-duo trend (add `--compare-rc-surface` for the GUI-built-app RC blocks); comparing across rulers requires re-scoring an archived transcript under the current ruler. **RC** = the release candidate a run scored (e.g. `v1.0.4-rc1`). -> Rows: **98** · rendered 2026-07-10T09:46:36+00:00 +> Rows: **100** · rendered 2026-07-10T13:36:04+00:00 | Run | When | SHA | Build date | Surface | DM model | Actor model | Scorer | Ruler | Lens ruler | RC | Methodology | Story | Mech | AngryDM | Behav | Sat | RRI | Crit | Img% | s/beat | cold-open s | turns/beat | combat s/beat | social s/beat | tool% | tool ms | slowest tool | Acts | Structural coverage | Engagement | Inert systems | Pass | Source | Notes | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| outdoor-lora-smoke-forest_road-2026-07-10 | 2026-07-10T13:36:04.551000+00:00 | f107e979 | | visual | | | sonnet-5 (5x independent blind scorers) | sc_38b768e2fd1b | lc_b031bd9f47e1 | | OUTDOOR-LORA smoke test (#1481): model_bfl-flux-1-dev depth-CN (str0.65/end0.6) + model_RsWEcQL2NWXwoyEodWVE2vWG (new 18-img exterior-only LoRA, loraScale0.7), seed7 anchor -> Gemini instruction-edit re-registration (structure-lock+dimetric-lock, referenceImages=anchor) over forest_road_greybox.png. Blind 3-image panel: CONTROL(poe2_control.jpg), cross-lane ANCHOR(camp_clearing_night_v2.jpg), CANDIDATE(fr_outdoorlora_registered.png). | | | | | | | | | | | | | | | | | | | | | FAIL | qa/evidence/plate-sprint/outdoor-lora/forest_road/ | SEVERE REGRESSION, NOT CONVERGED. 5/5 scorers independently flagged broken/melted/wireframe-box geometry ("disconnected wireframe boxes with doors/curtains", "nonsensical open-cubicle architecture") -- the new exterior-only LoRA still invents architecture on forest_road (mirrors ARM C interior-LoRA CB-FOREST finding, qa/evidence/.../cb_forest/panel_verdict.json, median 6.0 there vs median 2.0 here -- worse). Median 2.0 vs 6.0 baseline cap and >6.5 success bar: FAILS both. Root cause hypothesis: forest_road greybox represents trees as dense adjacent rectangular box volumes, which the model reads as architectural regardless of exterior-only training composition -- a geometry-shape bias, not purely a training-data-domain bias. Overlay+advisory edge-recall (qa/plate_overlays.py) misleadingly HIGH (0.9678) because the candidate preserved the blocky rectangular edges too literally (as walls, not trees) -- edge-recall is content-blind per #1491, visual overlay is the binding evidence. Cost: 4 anchor mints @9 CU + 1 Gemini pass, well under budget. Model: model_RsWEcQL2NWXwoyEodWVE2vWG. | +| outdoor-lora-smoke-camp-2026-07-10 | 2026-07-10T13:36:04.551000+00:00 | f107e979 | | visual | | | sonnet-5 (5x independent blind scorers) | sc_38b768e2fd1b | lc_b031bd9f47e1 | | OUTDOOR-LORA smoke test (#1481): model_bfl-flux-1-dev depth-CN (str0.65/end0.6) + model_RsWEcQL2NWXwoyEodWVE2vWG (new 18-img exterior-only LoRA, loraScale0.7), seed42 anchor -> Gemini instruction-edit re-registration (structure-lock+dimetric-lock, referenceImages=anchor) over camp_clearing_night_greybox.jpg. Blind 3-image panel: CONTROL(poe2_control.jpg), INCUMBENT(camp_clearing_night_v2.jpg, the actual adopted canonical_plate), CANDIDATE(camp_outdoorlora_registered.png). | | | | | | | | | | | | | | | | | | | | | | qa/evidence/plate-sprint/outdoor-lora/camp/ | BORDERLINE, modest improvement, NOT a clean pass. Candidate median 6.5 vs incumbent (camp_clearing_night_v2, the actual adopted canonical_plate) median 6.0 -- a +0.5 delta, sitting exactly AT the >6.5 success threshold (not clearly above it) and well short of the 7.0+ target. 3/5 scorers independently flagged a repeated concentric-ring/banding artifact in the dirt around the firepit ("artificial glow-radius artifact rather than natural ground texture") -- a real, consistent defect, not scorer noise. No invented architecture, no characters, no melted geometry -- generalizes cleanly here (unlike forest_road). Incumbent scorer notes (flat/repetitive cabin roof siding) independently reproduce its known backdrop_score=6.0 in room_recipes.json, cross-validating the blind methodology. Model: model_RsWEcQL2NWXwoyEodWVE2vWG. | | 1481-relight-on-styled-plate-panel-1 | 2026-07-10T09:06:00+00:00 | | | visual | | | sonnet | sc_38b768e2fd1b | lc_b031bd9f47e1 | | cohesion-probe relight A/B/C blind 5-scorer (character-in-scene integration) | | | | | | | | | | | | | | | | | | | | | | | Relight A/B on warm styled plate (#1481). This row = the RELIT arm B (median 2.5); A_flat 7.5, C_control 6.5 control_valid=false. Relight REGRESSES vs flat (unanimous last) on greybox-sidecar banding; verdict STOP relight. Evidence qa/evidence/1481/. | | plate-probe-placement-crypt-20260710-v2-20260710T090411Z | 2026-07-10T09:04:11+00:00 | | | visual | | | sonnet | sc_38b768e2fd1b | lc_b031bd9f47e1 | | plate_loop blind panel (visual-critic recipe, 5 scorers) | | | | | | | | | | | | | | | | | | | | | | evidence/plate-sprint/probe-placement | plate_loop probe-placement-crypt-20260710-v2 · room=crypt · registration recall=None (min 0.95, FAIL) · pregate=FLAG · panel medians A/B/C=7.0/8.0/8.0 · delta_vs_control=-1.0 · control_valid=True · Candidate (crypt cohesion frame, PROBE-PLACEMENT fix #1386) median 7.0 vs registered PoE2 control median 8.0 (delta -1.0, WITHIN the +/-1.2 panel noise band -> MEETS the bar) vs incumbent/house-best median 8.0 this panel (house_best is the disclosed unchanged reference file, so its own median is a same-file tell, not a live comparison). All 5/5 scorers independently confirmed the placement fix: 'no character is standing on the tomb lid', 'goblin correctly grounded on the floor tile beside the sarcophagus's front-left corner, not on the tomb lid'. Versus the PRE-FIX adopt-crypt panel on the same plate (candidate median 6.0 vs control 9.0, delta -3.0, with 3/5 scorers flagging actors on the sarcophagus), this is a +1.0-point gain AND a qualitative fix of the placement defect the panel exists to catch. | | plate-forest_road-armC-cb-forest-lora-anchor-20260710T085154Z | 2026-07-10T08:51:54+00:00 | | | visual | | | sonnet | sc_38b768e2fd1b | lc_b031bd9f47e1 | | plate_loop blind panel (visual-critic recipe, 5 scorers) | | | | | | | | | | | | | | | | | | | | | | /Users/lume/worldos-session-notes/plate-sprint/forest_road/cb_forest/plateloop | plate_loop forest_road-armC-cb-forest-lora-anchor · room=forest_road · registration recall=0.4519 (min 0.95, FAIL) · pregate=FLAG · panel medians A/B/C=6/9/6 · delta_vs_control=0 · control_valid=False · NOT CONVERGED. Candidate (A) median 6.0 -- ties the outdoor incumbent's own established bar (camp_clearing_night_v2, 6.0 in prior cross-lane panels) exactly, same as iter3's z-image-anchor bootstrap, and stays well below the >=7.0 sprint bar. ARM C's painterly-architectural LoRA does NOT lift the outdoor class: it does not improve panel score over iter3 (6.0==6.0), AND it measurably WORSENS registration fidelity (edge-recall 0.4519 vs iter3's 0.9902 on the identical base+prompt+recipe, advisory per #1491 but corroborated by the overlay showing an invented stone tunnel absent from the greybox) and introduces a unanimous, panel-flagged invented-architecture tell that iter3's anchor did not carry. The LoRA's 10/10 architectural-interior training set overpowers the depth-ControlNet + explicit anti-architecture prompt clause on this natural-exterior room. Verdict: ARM C is validated for its intended architectural/interior use case (per issue #1481) but does NOT generalize to, and actively regresses, the outdoor forest-road anchor-minting task versus the already-adopted z-image+house-LoRA layered-anchor bootstrap (iter3). |