One shot mode is no longer populating question tables with the assumptions

[acreeger]

This was likely a regression caused by either introduction of the hooks or when AskUserQuestion tool was added

[acreeger]

Bug Report Analysis

Questions for Reporter

Question	Answer
What specific command/flags trigger this behavior? (e.g., il start 123 --one-shot)	Assuming il start <issue> --one-shot based on README
What does "question tables" mean - are these markdown tables in agent comments?	Assuming markdown tables in iloom-issue-enhancer output with "Questions for Reporter" section
What should the "assumptions" column contain when populated correctly?	Assuming the Answer column should have agent's assumed answers instead of being blank
When did this last work correctly - before which commit/release?	Unknown - needs user confirmation
Does this affect only the enhancer agent, or other agents too (evaluator, etc.)?	Assuming only enhancer based on context, but needs confirmation

Problem Summary
When running iloom in one-shot mode, the issue enhancer agent no longer fills in the Answer column of question tables with its assumptions, leaving them blank when they should contain the agent's assumed responses.

User Impact
Users running one-shot mode lose visibility into what assumptions the AI agents are making during automated analysis, reducing transparency and making it harder to catch incorrect assumptions before implementation begins.

Expected Behavior
In one-shot mode, when the enhancer agent has questions for the reporter but must proceed without human input, it should populate the Answer column of question tables with its best assumptions.

Actual Behavior
The Answer column in question tables remains empty/blank in one-shot mode, providing no visibility into what assumptions the agent made.

Next Steps

• Reporter to confirm specific command, what "question tables" refers to, and when this last worked
• Technical analysis to compare hook system changes and AskUserQuestion tool introduction
• Identify which change broke the assumption-filling behavior
• Implementation planning and execution

---

📋 Complete Context & Details (click to expand)

Reproduction Steps (assumed - needs confirmation)

1. Run il start <issue-number> --one-shot on an issue with incomplete information
2. Wait for the enhancer agent to analyze the issue
3. Check the issue comment created by the enhancer agent
4. Look for the "Questions for Reporter" table
5. Observe that the Answer column is empty instead of containing assumed answers

Additional Context

From the issue description:

• This is suspected to be a regression
• Likely caused by either:
  • Introduction of the hooks system
  • Addition of the AskUserQuestion tool
• These were recent changes to the codebase

From README.md context:

• One-shot mode (--one-shot flag) automates the entire pipeline from start to finish without prompts
• The enhancer agent (iloom-issue-enhancer) expands brief issue descriptions into detailed requirements
• Agents write structured output to issue comments for permanent project knowledge
• The system should maintain transparency by documenting AI reasoning

Impact Analysis

• Transparency Loss: Users can't see what assumptions were made in automated mode
• Risk Increase: Incorrect assumptions may go unnoticed until implementation
• Workflow Inconsistency: Different behavior between interactive and one-shot modes
• Trust Erosion: Reduces confidence in automated workflow if reasoning is hidden

Technical Context (for investigation)
Areas to investigate:

1. Hook system changes - when were hooks introduced and how do they intercept agent interactions?
2. AskUserQuestion tool - does this tool have different behavior in one-shot vs interactive mode?
3. Agent prompt changes - was the instruction to fill assumption tables removed or changed?
4. Comment generation logic - are tables being generated but not populated, or not generated at all?

[acreeger]

Complexity Assessment

Classification: SIMPLE

Metrics:

• Estimated files affected: 2
• Estimated lines of code: 75
• Breaking changes: No
• Database migrations: No
• Cross-cutting changes: No
• File architecture quality: Good (well-organized templates with clear conditional logic)
• Overall risk level: Low

Reasoning: One-shot mode skips template sections that populate assumption tables. Fix requires adding fallback instructions in the non-interactive code path to ensure agents still document their assumptions, even without user validation. Changes are confined to template conditionals with low risk and no breaking changes.

[acreeger]

Combined Analysis & Plan - Issue #396

Executive Summary

The one-shot mode no longer populates question tables with assumptions due to a regression in commit 619b9eb. When INTERACTIVE_MODE was introduced, the assumption validation blocks were wrapped in {{#IF INTERACTIVE_MODE}} conditionals, but no equivalent {{#IF ONE_SHOT_MODE}} fallback was added. In one-shot mode, these blocks are simply removed, leaving agents without instructions to document their assumptions.

HIGH/CRITICAL Risks (if any)

None - this is a straightforward template fix with no architectural impact.

Implementation Overview

High-Level Execution Phases

1. Add ONE_SHOT_MODE conditional blocks to issue-prompt.txt alongside existing INTERACTIVE_MODE blocks
2. Add matching blocks to pr-prompt.txt for consistency
3. Verify template processing with existing tests

Quick Stats

• 2 files to modify
• 0 new files to create
• 0 files to delete
• Dependencies: None

---

Complete Analysis & Implementation Details (click to expand)

Research Findings

Problem Space

• Problem: One-shot mode agents don't receive instructions to populate assumption tables because the instructions are inside INTERACTIVE_MODE conditional blocks
• Architectural context: Template system uses mustache-like conditionals ({{#IF VAR}}...{{/IF VAR}}) to include/exclude sections based on mode
• Edge cases: None - both modes are mutually exclusive

Codebase Research

• Entry point: /templates/prompts/issue-prompt.txt - template processed by PromptTemplateManager
• Dependencies: src/lib/PromptTemplateManager.ts:324-333 handles INTERACTIVE_MODE conditionals
• Similar patterns: ONE_SHOT_MODE conditionals exist in PromptTemplateManager but are no longer used in issue-prompt.txt

Historical Context (Regression)

• Commit: 619b9eb - "Add interactive mode support to prompt template system. Fixes The enhance, analysis and planning agents should use the AskUserQuestion tool (if interactive and not one-shot mode) to ask clarifying questions during the process, instead of making assumptions. Agents should not ask questions that have already been asked and answered in previous comments #248"
• What changed: Agent execution instructions were enhanced to include detailed assumption validation workflows (steps 2.5, 3.5) wrapped in INTERACTIVE_MODE blocks
• Why it broke: The commit replaced simple "instruct agents to add their own answers" text with conditional INTERACTIVE_MODE blocks but added no ONE_SHOT_MODE alternative
• Before (working): "Execute: @agent... instructing them to add their own answers to any questions..."
• After (broken): "Execute: @agent... with the following instructions about assumptions..." followed by INTERACTIVE_MODE-only validation steps

Affected Files

• /templates/prompts/issue-prompt.txt:272-288 - STEP 1 Enhancement Phase - has INTERACTIVE_MODE block, needs ONE_SHOT_MODE fallback
• /templates/prompts/issue-prompt.txt:303-313 - SKIP_ENHANCEMENT path - needs one-shot handling
• /templates/prompts/issue-prompt.txt:322-338 - STEP 1.5 Complexity Evaluation - needs ONE_SHOT_MODE fallback
• /templates/prompts/issue-prompt.txt:390-406 - STEP 2 Analysis Phase - needs ONE_SHOT_MODE fallback
• /templates/prompts/issue-prompt.txt:468-484 - STEP 2-SIMPLE Combined Analysis - needs ONE_SHOT_MODE fallback
• /templates/prompts/issue-prompt.txt:508-524 - STEP 3 Planning Phase - needs ONE_SHOT_MODE fallback
• /templates/prompts/pr-prompt.txt - No conditional blocks currently exist, so no changes needed (it's a simpler workflow)

---

Implementation Plan

Automated Test Cases to Create

No new tests needed - existing tests in src/commands/ignite.test.ts already verify ONE_SHOT_MODE flag is passed correctly to the template manager.

Files to Modify

1. /templates/prompts/issue-prompt.txt

For each INTERACTIVE_MODE block (6 occurrences), add a corresponding ONE_SHOT_MODE block immediately before or after that provides simple instructions for autonomous assumption documentation.

Pattern for each occurrence:

{{#IF INTERACTIVE_MODE}}
[existing validation steps 2.5/3.5]
{{/IF INTERACTIVE_MODE}}
{{#IF ONE_SHOT_MODE}}
2.5. Document assumptions autonomously:
   - Any questions you have about the issue should be documented in a "Questions and Key Decisions" table
   - Provide your assumed answer in the Answer column based on context, code analysis, and best practices
   - These assumptions will be used to proceed without human validation
{{/IF ONE_SHOT_MODE}}

Specific changes:

Location 1: Lines 272-288 (STEP 1 Enhancement)
Add ONE_SHOT_MODE block after the existing INTERACTIVE_MODE block at line 288.

Location 2: Lines 303-313 (SKIP_ENHANCEMENT)
Add ONE_SHOT_MODE block after INTERACTIVE_MODE block at line 313 - this handles checking existing enhancement for unanswered questions.

Location 3: Lines 322-338 (STEP 1.5 Complexity Evaluation)
Add ONE_SHOT_MODE block after INTERACTIVE_MODE block at line 338.

Location 4: Lines 390-406 (STEP 2 Analysis Phase)
Add ONE_SHOT_MODE block after INTERACTIVE_MODE block at line 406.

Location 5: Lines 468-484 (STEP 2-SIMPLE Combined Analysis)
Add ONE_SHOT_MODE block after INTERACTIVE_MODE block at line 484.

Location 6: Lines 508-524 (STEP 3 Planning Phase)
Add ONE_SHOT_MODE block after INTERACTIVE_MODE block at line 524.

Detailed Execution Order

Phase 1: Update issue-prompt.txt

1. After each {{/IF INTERACTIVE_MODE}} block, add corresponding ONE_SHOT_MODE block with condensed assumption documentation instructions
2. Build project: pnpm build -> Verify: Compiles successfully
3. Manual test: Run il start <issue> --one-shot and verify agents create assumption tables with populated Answer columns

NOTE: Follow the project's development workflow as specified in CLAUDE.md.

Dependencies and Configuration

None

[acreeger]

Implementation Complete

Root Cause

Commit 619b9eb ("Add interactive mode support to prompt template system") changed the agent execution instructions from:

Original (proactive):

"instructing them to add their own answers to any questions they asked in the question tables they create in their issue comments"

Changed to (scanning-focused):

"Before making assumptions, scan ALL issue comments for previously asked questions and their answers. Document any assumptions clearly in question tables with your own answers."

The original instruction explicitly told agents to fill in answers to questions they create. The new wording focused on scanning existing content, losing the proactive instruction.

Fix Applied

Restored the proactive instruction by splitting into two clear directives (unconditional - applies to both interactive and one-shot modes):

1. Execute: @agent-iloom-issue-enhancer ISSUE_NUMBER with the following instructions about assumptions:
   - Before making assumptions, scan ALL issue comments for previously asked questions and their answers.
   - When creating question tables in your issue comments, fill in your own assumed answers to each question. This documents assumptions made during execution.

This benefits both modes:

• One-shot mode: Agents document assumptions for transparency
• Interactive mode: Users can quickly confirm pre-filled answers rather than starting from scratch

Files Modified

File	Change
templates/prompts/issue-prompt.txt	Updated 6 agent execution instructions to include proactive assumption-filling directive

Validation

• ✅ Tests: 3066 passed
• ✅ Build: Success
• ✅ Lint: Success

[acreeger]

Additional Implementation - PR Template

Working on applying the same {{#IF ONE_SHOT_MODE}} pattern to /templates/prompts/pr-prompt.txt

• Review pr-prompt.txt for INTERACTIVE_MODE blocks
• Verify no changes needed

Finding

The pr-prompt.txt template does not contain any {{#IF INTERACTIVE_MODE}} blocks - it's a simpler workflow template for existing PRs that doesn't include the multi-phase agent execution workflow (enhancement, analysis, planning phases) with assumption validation steps.

The original implementation plan correctly noted: "pr-prompt.txt - No conditional blocks currently exist, so no changes needed (it's a simpler workflow)"

No changes required to pr-prompt.txt. The fix in issue-prompt.txt is complete.

[acreeger]

iloom Session Summary

Key Themes:

• Regression in commit 619b9eb removed proactive instruction for agents to fill in assumption tables, breaking one-shot mode transparency
• Original instruction told agents to fill answers to questions they create; replacement only focused on scanning existing comments
• Fix restores unconditional proactive instruction, benefiting both interactive (faster confirmation) and one-shot (documented assumptions) modes

---

Session Details (click to expand)

Key Insights

• The prompt template system uses {{#IF INTERACTIVE_MODE}} and {{#IF ONE_SHOT_MODE}} conditionals to adapt agent behavior based on workflow mode
• Commit 619b9eb introduced INTERACTIVE_MODE blocks for assumption validation but removed the original unconditional instruction that told agents to pre-fill their own answers
• The distinction between "scan existing comments for answers" and "fill in your own answers to questions you create" is critical - agents need both instructions
• Pre-filling assumptions unconditionally (not just in one-shot mode) provides value in interactive mode by giving users concrete answers to confirm rather than starting from blank questions

Decisions Made

• Kept assumption pre-filling unconditional across all modes rather than gating it behind {{#IF ONE_SHOT_MODE}} - users can confirm pre-filled answers faster than filling them from scratch
• Split the instruction into two clear directives: (1) scan existing comments for prior answers, (2) fill in your own assumed answers when creating new question tables
• Applied the fix to all 6 agent execution points in the workflow (enhancer, complexity evaluator, analyzer, analyze-and-plan, planner, implementer)

Challenges Resolved

• Initial fix attempt added retroactive {{#IF ONE_SHOT_MODE}} verification blocks after agent execution, but this didn't solve the core problem - agents needed proactive instructions before creating tables, not post-hoc validation
• The regression was subtle: the new instruction mentioned "document assumptions" but lacked the explicit directive to "fill in your own assumed answers to each question" that told agents what to do

Lessons Learned

• When modifying prompt templates, verify that conditional blocks have appropriate fallbacks - adding {{#IF INTERACTIVE_MODE}} without a corresponding {{#IF ONE_SHOT_MODE}} or unconditional instruction can silently remove behavior
• Proactive instructions (tell agents what to do upfront) are more effective than retroactive validation (check what agents did after the fact)
• The git history revealed the original working instruction in commits 0bc48ab (added one-shot support) and 5e8518a (made it universal) before 619b9eb changed the wording

---

Generated with 🤖❤️ by iloom.ai