Skip to content

feat(cli): self-improving printed CLIs - default-on self-healing learn loop#3434

Open
mvanhorn wants to merge 23 commits into
mainfrom
feat/self-improving-learn-loop
Open

feat(cli): self-improving printed CLIs - default-on self-healing learn loop#3434
mvanhorn wants to merge 23 commits into
mainfrom
feat/self-improving-learn-loop

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Intent

Make every printed CLI self-improving, on by default. PR #2440 ported the self-teaching loop into the generator templates a month ago, but enablement was opt-in via a spec block no skill authored, so zero fresh prints carried it (0 of 37 local library CLIs had internal/learn/). And where the loop did exist, its capture side was passive: a real two-session ESPN incident showed the CLI observing every failure signal (an unknown --date flag, a World Cup league code absent from print-time seeds, phrasing-split query families, no playbook recorded) and doing nothing with any of it, so each session re-derived ~70s of identical discovery.

This turns the loop on for every fresh print AND makes it self-healing: the CLI journals its own invocations, derives correction candidates from its own failures, synthesizes playbooks from the discovery a session actually ran, and surfaces them for the agent to confirm - no agent-volunteered protocol required. Local-only measurement (learnings stats + a learn_events table) with pre-registered numeric kill criteria decides whether the loop earns its keep.

Approach

18 implementation units in one branch, flip landing last so no print is default-on before the quality bar is met:

  • Distribution: ApplyLearnLoopDefault at the Generate() entry only (never GenerateMCPSurface - mcp-sync must not flip learn onto published CLIs that lack the package); learn.disabled: true opt-out; store auto-promoted instead of hard-erroring; x-learn OpenAPI extension.
  • Self-healing capture: segmented-JSONL invocation journal, flag-correction derivation, teach-time playbook synthesis, quarantined learn_candidates lifecycle (confirm prints the payload before materializing; reject tombstones), recall envelope candidates[] with two-step byte-exact next-actions, same-referent phrasing folding with lazy rekey, entity-lookup refresh from synced data (the World Cup root cause).
  • Quality: typed exit codes on teach commands, honest mcp:local-write annotations plus a telemetry-class read-only carve-out in AGENTS.md, MCP context-tool protocol parity from one shared source, PII warn-scan at teach time, behavioral scorecard credit replacing the presence sentinel.
  • Measurement: learn_events (row-id joined), learnings stats, forget-cascades-events, 30-day kill-criteria clock baked into every generated AGENTS.md.
  • Schema: one additive bump to v9 (learn_candidates + learn_events); resourcesFTSContentSchemaVersion decoupled and pinned so reprints never trigger an FTS rebuild.
  • Skill: required Pre-Generation Learn Enrichment step in /printing-press so seeds get authored.

Verification

  • go test ./... green (full suite, unscoped).
  • scripts/golden.sh verify 31/31 (intentional full regen for default-on; new generate-learn-disabled-api case pins the opt-out shape).
  • scripts/verify-generator-output.sh 10/10, scripts/verify-learn-purity.sh 52/52, go vet clean.
  • Incident-replay harness (TestIncidentReplay) proves AE1-AE7 end to end against a generated CLI, including the session-B-invocations-<=-50%-of-session-A gate and the pid-lineage session-key fallback.
  • Acceptance print on testdata/loops.yaml (a genuinely thin, POST-heavy, forced-store API): the incident heals end to end - recall miss -> fail --agen -> correct --agent -> the CLI derives a flag_alias candidate with the right family anchor -> recall surfaces it with a two-step next-action -> learnings confirm materializes the correction into the family playbook notes; verify-env runs write zero events and zero journal segments; 8 quality gates pass.

Cross-repo lockstep

Tracking issue filed at mvanhorn/printing-press-library#1425 (schema/package-shape changes + sweep anchors) per the AGENTS.md learn-template lockstep rule. ESPN hand-port via regen-merge is a separate PR (the proving vehicle); the other ~35 published CLIs are deferred to the sweep.

AI / Automation Disclosure

  • Human-reviewed: AI or automation was used, and a human reviewed the work for intent, fit, and obvious issues before submission.

Plan (local, gitignored): docs/plans/2026-07-02-002-feat-self-improving-learn-loop-unified-plan.md

🤖 Generated with Claude Code

https://claude.ai/code/session_019LMUA4Az5S9hvnaPfDt4W1

mvanhorn added 20 commits July 2, 2026 11:56
…ilent events; harness proves the incident heals
@greptile-apps

greptile-apps Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Too many files changed for review. (157 files found, 100 file limit)

Bypass the limit by tagging @greptile-apps to review.

@mergify

mergify Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 2 of 2 protections blocking · waiting on 🤖 CI and 🙋 you

Protection Waiting on
🔴 require-ready-label-and-ci 🤖 CI and 🙋 you
🔴 🚦 Auto-queue 🙋 you

🔴 require-ready-label-and-ci

Waiting for

  • any of:
    • label = ready-to-merge
    • title ~= ^chore\(main\): release
  • any of:
    • check-neutral = Greptile Review
    • check-skipped = Greptile Review
    • check-success = Greptile Review
    • label = queued
    • title ~= ^chore\(main\): release
This rule is failing.
  • any of:
    • label = ready-to-merge
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release
  • any of:
    • check-neutral = Greptile Review
    • check-skipped = Greptile Review
    • check-success = Greptile Review
    • head ~= ^mergify/merge-queue/
    • label = queued
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release
  • #changes-requested-reviews-by = 0
  • #review-threads-unresolved = 0
  • check-success = build-and-test
  • check-success = generated-test
  • check-success = go-lint
  • check-success = golden
  • check-success = pr-title
  • check-success = test
  • any of:
    • -files ~= ^(\.github/workflows/|\.github/scripts/|scripts/|\.github/CODEOWNERS$)
    • author = mvanhorn
    • approved-reviews-by = mvanhorn
    • approved-reviews-by = tmchow
    • author = tmchow

🔴 🚦 Auto-queue

Waiting for

  • label = ready-to-merge
This rule is failing.

When all merge protections are satisfied and these conditions match, this pull request will be queued automatically.

  • label = ready-to-merge

mvanhorn and others added 3 commits July 3, 2026 16:04
… by edit distance

When a failed flag has no recorded did-you-mean suggestion, infer the correction
from the single new flag introduced by the successful retry (closest by edit
distance; ambiguous ties skipped). Cap acceptance at min(len(failed),
len(candidate))/2 (floor 1) so an unrelated new success flag (e.g. --sport when
--date failed) is not promoted as a spurious flag_alias. Adds positive and
negative test coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q4eADiHsstb6svS3iqsK4f
UpsertLearning/ListLearnings/ForgetLearnings and the teach.go call chain now
thread context.Context (BeginTx/QueryContext/ExecContext) so Ctrl-C can
interrupt teach writes and reads. This fix shipped hand-patched in a published
CLI but was never upstreamed into the templates; without it, a reprint under
the current press regresses ctx propagation. Regenerated learn-loop goldens.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q4eADiHsstb6svS3iqsK4f
Pre-existing findings surfaced once .golangci.yml adopted the modernize
linter: interface{} -> any, strings.Split -> strings.SplitSeq, and an
if/assign -> min. No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q4eADiHsstb6svS3iqsK4f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant