Draft-generator fails decline/hedge scenarios (dg-3 baseline 1/10)

[ankitvgupta]

Summary

The eval framework's dg-3-decline fixture surfaces a real product weakness: when an incoming email asks the user to commit to something they haven't decided about (e.g. a speaking invitation), the draft-generator consistently produces drafts that:

• Soft-accept on the user's behalf ("I'd probably be up for it!")
• Ask irrelevant questions (e.g. "what time of day?" when the email already states a fixed date)
• Commit the user to follow-up timelines without their authorization
• Almost never produce a clean decline OR a clean hedge

Two independent judges (claude-sonnet-4-6 and claude-opus-4-7) agree the feature is failing this scenario:

Judge	Mean	Median	Stddev
Sonnet 4.6	2.3/10	3/10	1.49
Opus 4.7	2.0/10	2/10	1.18

Both judges consistently cite the same failure modes. This is real signal, not eval noise.

Repro

npx tsx scripts/evals-variance.ts --runs 10 --concurrency 5 --feature draft-generator
# Look at dg-3-decline scores

Or open the running app, find an email that's an invitation (low priority, requires yes/no), click Generate Draft, observe the output.

The fixture

tests/evals/feature-fixtures/draft-generator/dg-3-decline.json — an invitation to a fixed-date speaking event at IndustryConf, asking the user to commit by next week.

Likely fix surface

src/shared/types.ts → DEFAULT_DRAFT_PROMPT (and the format suffix in src/main/services/draft-generator.ts). Probably needs explicit instructions for the "you don't know what to do" branch:

• If the email asks the user to commit to a decision the AI has no information about → produce a hedge that defers to the user (e.g., "Thanks for thinking of me. I'll get back to you after checking my calendar.")
• Do NOT soft-commit on the user's behalf with phrases like "probably," "likely," "should be able to"
• Do NOT ask clarifying questions about details that are already in the email
• For fixed-date invitations (panels, conferences, etc.), do NOT propose alternative dates

Acceptance criteria

dg-3-decline baseline score (Opus judge, 10-run mean) lifts from current ~2/10 to >= 7/10. Re-run scripts/evals-variance.ts --feature draft-generator and update the baseline accordingly.

Why we surfaced this now

This is exactly the kind of bug the LLM-judge eval framework is built to catch — a real quality regression that no single run would have flagged. The framework is doing its job; the prompt needs work.

Discovered while building #123 (12/10 testing suite).