Coding agent visual + runtime verification: screenshot + console errors + simulator/emulator testing

[joelteply]

Problem

The coding agent produced a Snake game that has runtime JavaScript errors and renders a blank canvas. Without verification, this becomes training data that teaches the model to produce broken code. Loss going down means NOTHING if the output doesn't work.

Current State

• Code generated: looks correct syntactically
• Runtime errors: Cannot read properties of undefined (reading 'set') line 345, Cannot read properties of undefined (reading 'length') line 147
• Visual result: blank black canvas (no snake, no food, no overlay)
• Training capture: captureTraining=true saved this as a POSITIVE example — it should be NEGATIVE

What's Needed

1. Browser Console Error Capture

After the coding agent opens a file in browser:

• Capture console.error and uncaught exceptions via jtag
• Inject a small error collector script before loading the page
• Return errors as structured data in the tool result

2. Screenshot Verification

• interface/screenshot of the rendered page
• VisionDescriptionService describes what it sees
• Compare description against the prompt requirements
• "Blank canvas" ≠ "Snake game with score display"

3. Automated Grading

• Runtime errors = automatic failure
• Visual mismatch = failure (blank when should be game)
• Passing = no errors + visual matches description
• Failed examples become negative training data (or are discarded)

4. Simulator/Emulator Testing (future)

For iOS/Android apps:

• iOS Simulator driven tests (xcrun simctl)
• Android Emulator driven tests (adb)
• Secure enclave testing for biometric features
• Screenshot capture from simulator/emulator
• Same visual verification pipeline

5. Fix-and-Retry Loop

When verification fails:

• Feed errors back to the coding agent
• "Your code has these errors: [errors]. The screenshot shows: [description]. Fix it."
• Retry up to N times
• Only successful, verified code becomes training data

The Rule

No blind training. Every code example must be:

1. Executed (not just compiled)
2. Visually verified (screenshot + description)
3. Error-free (no console errors, no crashes)
4. Functionally correct (output matches prompt)

Training on unverified code is training on garbage.

Related

• Sensory system verification: vision, screenshots, live mode visual awareness #409 (sensory system — visual verification)
• Academy: no full training session proven end-to-end #377 (Academy e2e — needs verification in the loop)
• Support <think> tags: reasoning display in chat + typing→thinking verb #440 ( tags — model reasoning about its own code quality)

[joelteply]

Interaction testing for Academy evaluation:

Beyond just loading + error capture, the teacher needs to INTERACT with the output:

Game testing:

• Load → screenshot start screen
• Press arrow key → screenshot gameplay
• Wait for food collision → screenshot score change
• Die → screenshot game over screen
• Each checkpoint graded: does it look right?

App testing:

• Load → screenshot initial state
• Click buttons → screenshot state changes
• Fill forms → submit → screenshot result
• Navigate tabs → screenshot each view

The pattern: Playwright actions (click, type, press, wait) + screenshots at key moments = visual test trace. The teacher reviews the trace, not just the code.

This is TDD for visual output. The test IS the interaction. The grade IS the screenshot comparison.

Playwright already supports all of this:

await page.click('#start-button');
await page.screenshot({path: 'after-click.png'});
await page.keyboard.press('ArrowRight');
await page.waitForTimeout(1000);
await page.screenshot({path: 'gameplay.png'});

The coding agent should WRITE these interaction tests as part of its output. The verify command runs them. The teacher sees the screenshots.

[joelteply]

Visual regression testing for avatar/3D quality:

Claude can screenshot but can't judge visual quality — corrupted vertices look like 'pixels' to an LLM. Need automated visual regression:

1. Capture known-good reference screenshots for each avatar tile
2. On each deploy/change, screenshot current state
3. Pixel diff (or perceptual hash diff) against reference
4. Threshold exceeded → flag as regression

Tools: Playwright + pixelmatch (npm), or Percy/Chromatic-style snapshot testing. This is standard visual regression testing applied to 3D avatar rendering.

This would have caught the vertex corruption automatically.