Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,38 @@ PROJECT_PINNED_CLAUDE_BIN ?=

research_loop:
PROJECT_PINNED_CLAUDE_BIN="$(PROJECT_PINNED_CLAUDE_BIN)" ./scripts/claude-research-loop.sh

# Update patterns from existing research reports using Claude Code
# Examples:
# make update_patterns_from_research
# make update_patterns_from_research PATTERN=action-selector-pattern
# make update_patterns_from_research TEMPLATE_LINK="https://.../TEMPLATE.md"
PATTERN ?=
CLAUDE_BIN ?=
CLAUDE_MODEL ?=
PATTERNS_DIR ?=
RESEARCH_DIR ?=
LOG_DIR ?=
LOOP_DELAY_SECONDS ?=
TEMPLATE_LINK ?=

update_patterns_from_research:
@if [ -n "$(PATTERN)" ]; then \
PATTERNS_DIR="$(PATTERNS_DIR)" \
RESEARCH_DIR="$(RESEARCH_DIR)" \
LOG_DIR="$(LOG_DIR)" \
CLAUDE_BIN="$(CLAUDE_BIN)" \
CLAUDE_MODEL="$(CLAUDE_MODEL)" \
LOOP_DELAY_SECONDS="$(LOOP_DELAY_SECONDS)" \
TEMPLATE_LINK="$(TEMPLATE_LINK)" \
./scripts/update-patterns-from-research.sh --pattern "$(PATTERN)"; \
else \
PATTERNS_DIR="$(PATTERNS_DIR)" \
RESEARCH_DIR="$(RESEARCH_DIR)" \
LOG_DIR="$(LOG_DIR)" \
CLAUDE_BIN="$(CLAUDE_BIN)" \
CLAUDE_MODEL="$(CLAUDE_MODEL)" \
LOOP_DELAY_SECONDS="$(LOOP_DELAY_SECONDS)" \
TEMPLATE_LINK="$(TEMPLATE_LINK)" \
./scripts/update-patterns-from-research.sh; \
fi
35 changes: 27 additions & 8 deletions patterns/abstracted-code-representation-for-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,29 @@ tags: [code-review, verification, abstraction, pseudocode, intent-based-review,

## Problem

Reviewing large volumes of AI-generated code line-by-line can be tedious, error-prone, and inefficient. Human reviewers are often more interested in verifying the high-level intent and logical correctness of changes rather than minute syntactic details if the generation process is trusted to some extent.
Reviewing AI-generated code line-by-line is time-intensive and cognitively demanding. Research shows developers prefer understanding *why* changes were made over *how* they were implemented—intent-level review is faster and more effective than syntax-level verification.

## Solution

Provide a higher-level, abstracted representation of code changes for human review, rather than (or in addition to) the raw code diff. This could include:
Provide abstracted representations of code changes for human review:

- **Pseudocode:** Representing the logic of the changes in a more human-readable, concise format.
- **Intent Summaries:** Describing what the changes aim to achieve at a functional level.
- **Logical Diffs:** Highlighting changes in program behavior or structure rather than just textual differences.
- **Visualizations:** Graphical representations of control flow or data flow changes.
- **Pseudocode:** Concise, human-readable representation of logic
- **Intent Summaries:** Functional description of what changes achieve
- **Logical Diffs:** Behavioral changes rather than textual differences
- **Visualizations:** Control flow, data flow, or architectural diagrams

Crucially, this abstracted representation must come with strong guarantees (or at least high confidence) that it accurately and faithfully maps to the actual low-level code modifications that will be implemented. This allows reviewers to focus on conceptual correctness, significantly speeding up the verification process.
**Critical requirement:** Abstracted representations must have strong guarantees that they accurately map to actual code changes. Formal verification of this mapping remains an open research challenge; current implementations rely on confidence scoring and drill-down capability for verification.

**Production examples:** GitHub Copilot Workspace (multi-stage workflows), Cursor AI (intent-based editing), Claude Code (plan-then-execute verification), PR summarization tools (Augment: 59% F-Score, Cursor Bugbot: 49%, Greptile: 45%, CodeRabbit: 39%, Claude Code: 31%, GitHub Copilot: 25%).

## Example

Instead of reviewing 50 lines of Python implementing a new sorting algorithm, review:
"Changed sorting logic for `user_list` from bubble sort to quicksort to improve performance for large lists. Test coverage maintained."
With a system guarantee that this change is correctly implemented in the underlying Python code.

With drill-down capability to verify the underlying Python code matches the abstraction.

**Enterprise impact:** Tekion achieved 60% faster merge times with intent-based summaries; Microsoft reviews 600K+ PRs/month using AI-assisted abstraction (13.6% fewer errors); Tencent reported 68% decrease in production incidents.

## How to use it

Expand All @@ -45,3 +50,17 @@ With a system guarantee that this change is correctly implemented in the underly
- Aman Sanger (Cursor, referencing Michael Grinich) at 0:09:48: "...operating in a different representation of the codebase. So maybe it looks like pseudo code. And if you can represent changes in this really concise way and you have guarantees that it maps cleanly onto the actual changes made in the in the real software, that just shorten the time of verification a ton."

- Primary source: https://www.youtube.com/watch?v=BGgsoIgbT_Y

- Alon et al. (POPL 2019): code2vec—learning distributed representations of code via AST path-based embeddings

- Feng et al. (EMNLP 2020): CodeBERT—bimodal pre-training for programming and natural languages

- Buse & Weimer (FSE 2010): "What Did They Change?"—developers prefer intent-level understanding over implementation details

- Storey et al. (IEEE TSE 2002): Software visualization improves program comprehension through multiple abstraction views

- Bayer et al. (ICSE 2017): Modern Code Review at Google—reviewers focus on logical correctness over implementation details

- Zhang et al. (arXiv 2026): EyeLayer—human attention patterns improve code summarization quality

- Schäfer et al. (ICSE 2020): Semantic Differencing for Software Refactoring—behavioral vs. textual changes
6 changes: 5 additions & 1 deletion patterns/action-caching-replay.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ This creates several issues:

Record every action during execution with precise metadata (XPaths, frame indices, execution details), enabling deterministic replay without LLM calls. The cache captures enough information to replay actions even when page structure changes slightly.

This pattern builds on **experience replay** from reinforcement learning, where agents learn by reusing past successful actions rather than exploring anew each time.

### Core Approach

**Action cache entries** store complete execution metadata:
Expand Down Expand Up @@ -218,7 +220,7 @@ npx hyperagent script workflows/login-cache.json > login.test.ts

**Pros:**

- **Dramatic cost reduction**: Replay costs near-zero (no LLM calls) if XPaths work
- **Dramatic cost reduction**: Replay costs near-zero (no LLM calls) if XPaths work; documented cost reductions range from 43-97% across implementations; cache hit rates of 85%+ indicate excellent effectiveness
- **Deterministic regression testing**: Verify fixes don't break existing workflows
- **Performance**: Cached replays are 10-100x faster than LLM execution
- **Debugging**: Cache provides complete execution history
Expand All @@ -244,4 +246,6 @@ npx hyperagent script workflows/login-cache.json > login.test.ts

- [HyperAgent GitHub Repository](https://github.com/hyperbrowserai/HyperAgent) - Original implementation
- [HyperAgent Documentation](https://docs.hyperbrowser.ai/hyperagent/introduction) - Usage guide
- [Cost-Efficient Serving of LLM Agents via Test-Time Plan Caching](https://arxiv.org/abs/2506.14852) (Zhang et al., 2025) - Academic foundation showing 46.62% average cost reduction
- [Docker Cagent](https://github.com/docker/cagent) - Proxy-and-cassette model for deterministic agent testing
- Related patterns: [Structured Output Specification](structured-output-specification.md), [Schema Validation Retry](schema-validation-retry-cross-step-learning.md)
19 changes: 16 additions & 3 deletions patterns/action-selector-pattern.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ tags: [prompt-injection, control-flow, safety, tool-use]

## Problem

In tool-enabled agents, untrusted data from emails, web pages, and API responses is often fed back into the model between steps. That creates a control-flow vulnerability: injected text can influence which action the agent chooses next, not just what it writes. Even if individual tools are safe, a compromised action-selection loop can trigger harmful sequences at the orchestration layer.
In tool-enabled agents, untrusted data from emails, web pages, and API responses is often fed back into the model between steps. That creates a control-flow vulnerability: injected text can influence which action the agent chooses next, enabling control-flow hijacking. Even if individual tools are safe, a compromised action-selection loop can trigger harmful sequences at the orchestration layer and enable cascading prompt injection attacks.

## Solution

Expand All @@ -21,17 +21,26 @@ Treat the LLM as an instruction decoder, not a live controller. The model maps u
- Prevent tool outputs from re-entering the selector prompt.
- For multi-step workflows, compose actions in code with explicit state transitions.

This preserves natural-language usability while removing post-selection prompt-injection leverage.
This preserves natural-language usability while removing post-selection prompt-injection leverage. By preventing tool outputs from re-entering the LLM context, the pattern provides provable resistance to prompt injection through separation of duties and input/output control.

```pseudo
action = LLM.translate(prompt, allowlist)
execute(action)
# tool output NOT returned to LLM
```

## Evidence

- **Evidence Grade:** `high` (academically grounded; industry adoption confirmed)
- **Most Valuable Findings:**
- Provides provable resistance to control-flow hijacking via separation of duties and no feedback loop
- Supported by major frameworks: LangChain (tool allowlists, Pydantic validation), Anthropic Claude (function calling with response schemas), OpenAI (function calling with JSON Schema)
- Does NOT protect against parameter poisoning—malicious data can still influence parameters passed to approved tools
- **Unverified:** Detailed quantitative evaluation results from source paper

## How to use it

Provide a hard allowlist of actions (API calls, SQL templates, page links) and version it like an API contract. Use it for customer-service bots, routing assistants, kiosk flows, and approval systems where allowed actions are finite and auditable.
Provide a hard allowlist of actions (API calls, SQL templates, page links) and version it like an API contract. Use strict schema validation (e.g., Pydantic, JSON Schema) for all parameters. Use it for customer-service bots, routing assistants, kiosk flows, and approval systems where allowed actions are finite and auditable.

## Trade-offs

Expand All @@ -43,3 +52,7 @@ Provide a hard allowlist of actions (API calls, SQL templates, page links) and v
* Beurer-Kellner et al., §3.1 (1) Action-Selector.

- Primary source: https://arxiv.org/abs/2506.08837
- "ReAct" (Yao et al., 2022): Foundational reasoning-acting pattern that Action-Selector secures against injection
- SecAlign (Chen et al., 2024): Preference optimization defense against prompt injection
- StruQ (Chen et al., 2024): Structured query defense with type-safe construction
- "Learning From Failure" (Wang et al., 2024): Categories of tool-use errors in LLM agents
14 changes: 10 additions & 4 deletions patterns/adaptive-sandbox-fanout-controller.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@ Parallel sandboxes are intoxicating: you can spawn 10... 100... 1000 runs. But t
1. **Diminishing returns:** After some N, you're mostly paying for redundant failures or near-duplicate solutions
2. **Prompt fragility:** If the prompt is underspecified, scaling N just scales errors (lots of sandboxes fail fast)
3. **Resource risk:** Unbounded fan-out can overwhelm budgets, rate limits, or queues
4. **Oscillation risk:** Poorly tuned thresholds can cause scale-up/scale-down thrashing as the controller oscillates between decisions

Static "N=10 always" policies don't adapt to task difficulty, model variance, or observed failure rates.
Static "N=10 always" policies don't adapt to task difficulty, model variance, or observed failure rates. Most implementations use static caps rather than true signal-driven adaptation.

## Solution

Expand All @@ -41,6 +42,8 @@ Add a controller that *adapts fan-out in real time* based on observed signals fr

4. **Budget guardrails:** Enforce max sandboxes, max runtime, and "no-progress" stop conditions

5. **Hysteresis for stability:** Use different thresholds for scale-up vs. stop (e.g., scale up if confidence < 0.65, stop only if > 0.75) to prevent oscillation

```mermaid
flowchart TD
A[Task] --> B[Launch small batch N=3-5]
Expand Down Expand Up @@ -68,6 +71,7 @@ Concrete heuristics (example):
- Start N=3
- If >=2 succeed but disagree and judge confidence < 0.65 -> add +3 more
- If 0 succeed and top error signature covers >70% runs -> run a "spec clarifier" step, then restart
- **Hysteresis:** Stop only if confidence > 0.75 (higher threshold than scale-up) to prevent thrash
- Hard cap: N_max (e.g., 50), runtime cap, and "two refinement attempts then decompose"

## Trade-offs
Expand All @@ -81,11 +85,13 @@ Concrete heuristics (example):
**Cons:**

- Requires instrumentation (collecting failure signatures, confidence, diversity)
- Needs careful defaults to avoid oscillation (scale up/down thrash)
- Needs careful defaults and hysteresis to avoid oscillation (scale up/down thrash)
- Bad scoring functions can cause premature stopping
- Few verified implementations; most systems use static caps instead of true signal-driven adaptation

## References

* [Labruno: Scaling number of parallel sandboxes + judging winners (video)](https://www.youtube.com/watch?v=zuhHQ9aMHV0)
* [Labruno (GitHub)](https://github.com/nibzard/labruno-agent)
* [Labruno: Scaling number of parallel sandboxes + judging winners (video)](https://www.youtube.com/watch?v=zuhHQ9aMHV0) — **Note: Uses static `MAX_SANDBOXES` rather than true signal-driven adaptation**
* [Labruno (GitHub)](https://github.com/nibzard/labruno-agent) — Parallel execution with post-hoc judging, not adaptive fanout
* [OpenClaw Orchestrator](https://github.com/zeynepyorulmaz/openclaw-orchestrator) — Closest verified implementation; LLM decides next steps based on accumulated results
* Related patterns: [Swarm Migration Pattern](swarm-migration-pattern.md) (batch tuning, resource caps), [Sub-Agent Spawning](sub-agent-spawning.md) (switch to decomposition when needed)
33 changes: 28 additions & 5 deletions patterns/agent-assisted-scaffolding.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@ This allows developers to:
- Focus on the core logic rather than repetitive setup tasks.
- Ensure consistency in initial project structure.

**Scaffolding Modes:**

- **Text-to-code:** Natural language descriptions generate code structure
- **Design-to-code:** Figma, PSD, or design sketches convert to layouts (tools achieve ~92% layout accuracy)
- **Repository-aware:** Agents read existing codebases to scaffold compatible structures

**Critical for Future AI Agent Work**: The scaffolded structure becomes crucial context for subsequent AI agent interactions. Well-structured scaffolding with clear file organization, naming conventions, and architectural patterns helps future agents understand the codebase layout and make more informed decisions when implementing features or making modifications.

The agent acts as a "kickstarter" for new development efforts while simultaneously enriching the repository's structural context for future AI-assisted development.
Expand All @@ -37,17 +43,34 @@ flowchart TD

## How to use it

- Use this when humans and agents share ownership of work across handoffs.
- Start with clear interaction contracts for approvals, overrides, and escalation.
- Capture user feedback in structured form so prompts and workflows can improve.
**Best suited for:**

- New feature or module development
- Greenfield projects and prototyping
- Standardized frameworks (React, Express, etc.)
- Repetitive boilerplate generation

**Less effective for:**

- Legacy system integration (10+ year-old codebases)
- Highly regulated environments with strict compliance
- Complex business logic requiring deep domain expertise

**Core practice:** "AI scaffolds, you refine details"—review generated code at checkpoints before proceeding.

## Trade-offs

* **Pros:** Creates clearer human-agent handoffs and better operational trust.
* **Cons:** Needs explicit process design and coordination across teams.
* **Pros:** Faster time to first code, consistent project structure, reduced boilerplate.
* **Cons:** Code reliability issues (36% of developers report problems), requires human review, struggles with legacy integration. Scaffolding is essential—without it, configurations lead to "massive overengineering" (SANER 2026).

## References

- Lukas Möller (Cursor) mentions this at 0:03:40: "So I think for like initially laying out some code base, some new feature, it's very, very useful to just like use the agent feature to kind of get that started."

- Primary source: https://www.youtube.com/watch?v=BGgsoIgbT_Y

- "Biscuit: Scaffolding LLM-Generated Code" (2024). arXiv:2404.07387v1 - Explores scaffolding users to guide code generation and trust in AI-powered tools.

- "Scratch Copilot: Supporting Youth Creative Coding with AI" (2025). arXiv:2505.03867v1 - Implements supportive scaffolding mechanisms for real-time ideation, code generation, and debugging.

- "App.build: Scaffolding Environment-Aware Multi-Agent Systems" (2026). SANER 2026 Industrial Track, arXiv:2509.03310v2 - Ablation studies show configurations without scaffolding lead to "massive overengineering."
Loading
Loading