Hierarchical evaluation improvements - Presidio Evaluator v-next

.github/agents/Ralph.agent.md

@@ -0,0 +1,119 @@
+---
+name: Ralph
+description: Describe what this custom agent does and when to use it.
+argument-hint: The inputs this agent expects, e.g., "a task to implement" or "a question to answer".
+disable-model-invocation: false
+user-invocable: true
+---
+# Ralph Agent Instructions
+
+You are an autonomous coding agent working on a software project.
+
+## Your Task
+
+1. Read the PRD at `.ralph/prd.json` (from the repo root)
+2. Read the progress log at `.ralph/progress.txt`. If missing, create a new `.ralph/progress.txt` with a Codebase Patterns section.
+3. Read Codbase Patterns section in `AGENTS.md` and `copilot-instructions.md` under the repo root. If `AGENTS.MD` is missing, create one with an empty Codebase Patterns section.
+4. Check you're on the correct branch from PRD `branchName`. If not, check it out from the current checked-out branch (HEAD).
+5. Pick the **highest priority** user story where `passes: false`
+6. Implement that single user story
+7. Load the `quality-checks` skill and follow its instructions to verify the story before committing. If any check fails, fix the issues and retry until all checks pass.
+8. If all quality checks pass: Update the PRD to set `passes: true`, append progress to `.ralph/progress.txt` and commit ALL changes with message: `feat: [Story ID] - [Story Title]`
+9. If any of the hooks fail: Fix the issues and retry. Do NOT mark the story as passing.
+10. Update AGENTS.md files if you discover reusable patterns (see below)
+11. When finished, compare your implementation with the PRD to make sure the success criteria are met.
+
+## Progress Report Format
+
+APPEND to `.ralph/progress.txt` (never replace, always append):
+
+```
+## [Date/Time] - [Story ID]
+- What was implemented
+- Files changed
+- **Learnings for future iterations:**
+  - Patterns discovered (e.g., "this codebase uses X for Y")
+  - Gotchas encountered (e.g., "don't forget to update Z when changing W")
+  - Useful context (e.g., "the evaluation panel is in component X")
+---
+```
+
+The learnings section is critical - it helps future iterations avoid repeating mistakes and understand the codebase better.
+
+## Consolidate Patterns
+
+If you discover a **reusable pattern** that future iterations should know, add it to the repo root `AGENTS.md` file (create it if it doesn't exist). This file should consolidate the most important learnings:
+
+```
+## Codebase Patterns
+- Example: Use `sql<number>` template for aggregations
+- Example: Always use `IF NOT EXISTS` for migrations
+- Example: Export types from actions.ts for UI components
+- Always use `uv run` for Python — never bare python/pip
+```
+
+Only add patterns that are **general and reusable**, not story-specific details.
+
+## Update AGENTS.md Files
+
+Before committing, check if any edited files have learnings worth preserving in nearby AGENTS.md files:
+
+1. **Identify directories with edited files** - Look at which directories you modified
+2. **Check for existing AGENTS.md** - Look for AGENTS.md in those directories or parent directories
+3. **Add valuable learnings** - If you discovered something future developers/agents should know:
+   - API patterns or conventions specific to that module
+   - Gotchas or non-obvious requirements
+   - Dependencies between files
+   - Testing approaches for that area
+   - Configuration or environment requirements
+
+**Examples of good AGENTS.md additions:**
+
+- "When modifying X, also update Y to keep them in sync"
+- "This module uses pattern Z for all API calls"
+- "Tests require the dev server running on PORT 3000"
+- "Field names must match the template exactly"
+
+**Do NOT add:**
+
+- Story-specific implementation details
+- Temporary debugging notes
+- Information already in progress.txt
+- Implementation details that are too specific or likely to change
+
+Only update AGENTS.md if you have **genuinely reusable knowledge** that would help future work in that directory.
+
+## Quality Requirements
+
+- ALL commits must pass your project's quality checks (typecheck, lint, test)
+- Do NOT commit broken code
+- Keep changes focused and minimal
+- Follow existing code patterns
+- Validate the alignment between the PRD and the implementation when you're finished implementing
+
+## Browser Testing (Required for Frontend Stories)
+
+For any story that changes UI, you MUST verify it works in the browser:
+
+1. Load the `dev-browser` skill
+2. Navigate to the relevant page
+3. Verify the UI changes work as expected
+4. Take a screenshot if helpful for the progress log
+
+A frontend story is NOT complete until browser verification passes.
+
+## Stop Condition
+
+After completing a user story, check if ALL stories in `.ralph/prd.json` have `passes: true`.
+
+If ALL stories are complete and passing, reply with:
+<promise>COMPLETE</promise>
+
+If there are still stories with `passes: false`, end your response normally (another iteration will pick up the next story).
+
+## Important
+
+- Work on ONE story per iteration
+- Commit frequently
+- Keep CI green
+- Read the Codebase Patterns section in root `AGENTS.md` Codebase Patterns section and the `copilot-instruction.md` docs before starting

.github/instructions/convert-prd.md

@@ -0,0 +1,258 @@
+---
+name: ralph
+description: "Convert PRDs to prd.json format for the Ralph autonomous agent system. Use when you have an existing PRD and need to convert it to Ralph's JSON format. Triggers on: convert this prd, turn this into ralph format, create prd.json from this, ralph json."
+user-invocable: true
+---
+
+# Ralph PRD Converter
+
+Converts existing PRDs to the prd.json format that Ralph uses for autonomous execution.
+
+---
+
+## The Job
+
+Take a PRD (markdown file or text) and convert it to `prd.json` in your ralph directory.
+
+---
+
+## Output Format
+
+```json
+{
+  "project": "[Project Name]",
+  "branchName": "ralph/[feature-name-kebab-case]",
+  "description": "[Feature description from PRD title/intro]",
+  "userStories": [
+    {
+      "id": "US-001",
+      "title": "[Story title]",
+      "description": "As a [user], I want [feature] so that [benefit]",
+      "acceptanceCriteria": [
+        "Criterion 1",
+        "Criterion 2",
+        "Typecheck passes"
+      ],
+      "priority": 1,
+      "passes": false,
+      "notes": ""
+    }
+  ]
+}
+```
+
+---
+
+## Story Size: The Number One Rule
+
+**Each story must be completable in ONE Ralph iteration (one context window).**
+
+Ralph spawns a fresh Amp instance per iteration with no memory of previous work. If a story is too big, the LLM runs out of context before finishing and produces broken code.
+
+### Right-sized stories:
+- Add a database column and migration
+- Add a UI component to an existing page
+- Update a server action with new logic
+- Add a filter dropdown to a list
+
+### Too big (split these):
+- "Build the entire dashboard" - Split into: schema, queries, UI components, filters
+- "Add authentication" - Split into: schema, middleware, login UI, session handling
+- "Refactor the API" - Split into one story per endpoint or pattern
+
+**Rule of thumb:** If you cannot describe the change in 2-3 sentences, it is too big.
+
+---
+
+## Story Ordering: Dependencies First
+
+Stories execute in priority order. Earlier stories must not depend on later ones.
+
+**Correct order:**
+1. Schema/database changes (migrations)
+2. Server actions / backend logic
+3. UI components that use the backend
+4. Dashboard/summary views that aggregate data
+
+**Wrong order:**
+1. UI component (depends on schema that does not exist yet)
+2. Schema change
+
+---
+
+## Acceptance Criteria: Must Be Verifiable
+
+Each criterion must be something Ralph can CHECK, not something vague.
+
+### Good criteria (verifiable):
+- "Add `status` column to tasks table with default 'pending'"
+- "Filter dropdown has options: All, Active, Completed"
+- "Clicking delete shows confirmation dialog"
+- "Typecheck passes"
+- "Tests pass"
+
+### Bad criteria (vague):
+- "Works correctly"
+- "User can do X easily"
+- "Good UX"
+- "Handles edge cases"
+
+### Always include as final criterion:
+```
+"Typecheck passes"
+```
+
+For stories with testable logic, also include:
+```
+"Tests pass"
+```
+
+### For stories that change UI, also include:
+```
+"Verify in browser using dev-browser skill"
+```
+
+Frontend stories are NOT complete until visually verified. Ralph will use the dev-browser skill to navigate to the page, interact with the UI, and confirm changes work.
+
+---
+
+## Conversion Rules
+
+1. **Each user story becomes one JSON entry**
+2. **IDs**: Sequential (US-001, US-002, etc.)
+3. **Priority**: Based on dependency order, then document order
+4. **All stories**: `passes: false` and empty `notes`
+5. **branchName**: Derive from feature name, kebab-case, prefixed with `ralph/`
+6. **Always add**: "Typecheck passes" to every story's acceptance criteria
+
+---
+
+## Splitting Large PRDs
+
+If a PRD has big features, split them:
+
+**Original:**
+> "Add user notification system"
+
+**Split into:**
+1. US-001: Add notifications table to database
+2. US-002: Create notification service for sending notifications
+3. US-003: Add notification bell icon to header
+4. US-004: Create notification dropdown panel
+5. US-005: Add mark-as-read functionality
+6. US-006: Add notification preferences page
+
+Each is one focused change that can be completed and verified independently.
+
+---
+
+## Example
+
+**Input PRD:**
+```markdown
+# Task Status Feature
+
+Add ability to mark tasks with different statuses.
+
+## Requirements
+- Toggle between pending/in-progress/done on task list
+- Filter list by status
+- Show status badge on each task
+- Persist status in database
+```
+
+**Output prd.json:**
+```json
+{
+  "project": "TaskApp",
+  "branchName": "ralph/task-status",
+  "description": "Task Status Feature - Track task progress with status indicators",
+  "userStories": [
+    {
+      "id": "US-001",
+      "title": "Add status field to tasks table",
+      "description": "As a developer, I need to store task status in the database.",
+      "acceptanceCriteria": [
+        "Add status column: 'pending' | 'in_progress' | 'done' (default 'pending')",
+        "Generate and run migration successfully",
+        "Typecheck passes"
+      ],
+      "priority": 1,
+      "passes": false,
+      "notes": ""
+    },
+    {
+      "id": "US-002",
+      "title": "Display status badge on task cards",
+      "description": "As a user, I want to see task status at a glance.",
+      "acceptanceCriteria": [
+        "Each task card shows colored status badge",
+        "Badge colors: gray=pending, blue=in_progress, green=done",
+        "Typecheck passes",
+        "Verify in browser using dev-browser skill"
+      ],
+      "priority": 2,
+      "passes": false,
+      "notes": ""
+    },
+    {
+      "id": "US-003",
+      "title": "Add status toggle to task list rows",
+      "description": "As a user, I want to change task status directly from the list.",
+      "acceptanceCriteria": [
+        "Each row has status dropdown or toggle",
+        "Changing status saves immediately",
+        "UI updates without page refresh",
+        "Typecheck passes",
+        "Verify in browser using dev-browser skill"
+      ],
+      "priority": 3,
+      "passes": false,
+      "notes": ""
+    },
+    {
+      "id": "US-004",
+      "title": "Filter tasks by status",
+      "description": "As a user, I want to filter the list to see only certain statuses.",
+      "acceptanceCriteria": [
+        "Filter dropdown: All | Pending | In Progress | Done",
+        "Filter persists in URL params",
+        "Typecheck passes",
+        "Verify in browser using dev-browser skill"
+      ],
+      "priority": 4,
+      "passes": false,
+      "notes": ""
+    }
+  ]
+}
+```
+
+---
+
+## Archiving Previous Runs
+
+**Before writing a new prd.json, check if there is an existing one from a different feature:**
+
+1. Read the current `prd.json` if it exists
+2. Check if `branchName` differs from the new feature's branch name
+3. If different AND `progress.txt` has content beyond the header:
+   - Create archive folder: `archive/YYYY-MM-DD-feature-name/`
+   - Copy current `prd.json` and `progress.txt` to archive
+   - Reset `progress.txt` with fresh header
+
+**The ralph.sh script handles this automatically** when you run it, but if you are manually updating prd.json between runs, archive first.
+
+---
+
+## Checklist Before Saving
+
+Before writing prd.json, verify:
+
+- [ ] **Previous run archived** (if prd.json exists with different branchName, archive it first)
+- [ ] Each story is completable in one iteration (small enough)
+- [ ] Stories are ordered by dependency (schema to backend to UI)
+- [ ] Every story has "Typecheck passes" as criterion
+- [ ] UI stories have "Verify in browser using dev-browser skill" as criterion
+- [ ] Acceptance criteria are verifiable (not vague)
+- [ ] No story depends on a later story