Make the CLI work the way an agent expects it to - #18
Merged
Conversation
Four Haiku-class agents were pointed at this tool cold, with CLAUDE.md, AGENTS.md and docs/ withheld, and asked to do ordinary things: extract structured data, write a song, diagnose a broken file, drive the library. The point was to find out what the tool is like to use without alignment, since agents are most of who uses it. Two findings were worth acting on. Global flags only worked before the subcommand. `plainsong info song.song --json` was refused outright, and the refusal aborts the command. One agent hit it three times in one session -- --json, -v and --quiet -- and named flag position first when asked what it wished the tool had told it up front. This was already known and already documented; the documentation is what tools/verify_release.py was written against when it reported two false failures. Documenting a trap does not remove it, so all 24 subcommands now take the flags themselves. argparse.SUPPRESS is the load-bearing part: without it a subparser default overwrites the global value, so `plainsong --json info x` would parse and then quietly lose its --json. A loud failure traded for a silent one is the worse trade. `info` keeps its own --verbose rather than having it redefined. And info-level diagnostics were displayed by nothing at all -- not compile, not check, not `info --verbose`, which documents itself as showing every diagnostic. A whole severity was write-only. That is why `Title: My Song`, the most natural way to name a piece and not the notation, dropped the title silently, turned the row into a phantom section, and left `(untitled)` and a section count one too high as the only evidence. A fourth agent hit it and had to read parser.py to find out why. --verbose now includes them, in their own voice rather than painted as warnings; default output is unchanged and the corpus emits none. One thing the probes cleared rather than condemned: the library API. An agent found compile_text on its first read of __init__.py with no wrong turns, and the diagnostics on a deliberately broken file named all six faults explicitly. Those needed no work, which is worth recording as much as the defects. Both guards were mutation-tested: removing the subparser flags fails 24 of the new assertions, and reverting the verbose filter fails the other. Verified in each CI environment -- stdlib-only 702 tests 0 errors, pytest clean, specs 7/7, wheel 10/10, ruff clean, corpus fingerprint unmoved at 6,321 files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PBAjxy7cD6DzJ72NX8TJEc
Six agents were pointed at this CLI cold. The split was clean: every one that reached `plainsong new` succeeded, and the two that hand-authored blind hit the traps. One reached for `new` even after being told not to, then read the format off the file it generated -- which is the strongest evidence available that scaffolding is the path agents actually take, and that it works. The failures were on the other path: adding a file to a project that already exists, where there is nothing to scaffold from. There the format has to be guessed, and the guess was `Title:`, which is not the notation and which cost a dropped title and a phantom section. One agent asked for exactly this fix -- that `--help` show the shape so it need not generate a file to learn it. So `--help` carries the notation now. That makes it a claim about the language rather than decoration, so it is held to the bar this repository holds prose to: the sample must compile, must sound, and must produce no diagnostics. Replacing `**TRACK:**` with `Title:` in it fails two of the three guards. Worth recording honestly: neither follow-up probe exercised the two fixes in the previous commit. Both took the scaffold path and both used the old global flag position, so they are evidence of no regression and nothing more. Those fixes rest on the direct before/after and their mutation tests, not on these runs. 705 tests, specs 7/7, ruff clean, corpus fingerprint unmoved, and `check` over all 6,337 sources reports the same 4 warnings it reports on master. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PBAjxy7cD6DzJ72NX8TJEc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agents are most of who uses this tool, so rather than reason about what they find hard, I measured it: six Haiku-class agents were pointed at the CLI cold, with
CLAUDE.md,AGENTS.mdanddocs/explicitly withheld, and asked to do ordinary things — extract structured data, write a song, diagnose a broken file, drive the library.What the tool already does well
Two probes cleared it outright, and that's worth as much as the defects:
compile_texton its first read of__init__.py, made zero wrong guesses, hit noAttributeErrororTypeError. Asked whether there was an obvious front door: "Absolutely yes."The split that explains everything else
plainsong new, or an existing fileTitle:plainsong newis the strongest affordance here — one agent reached for it even after being told not to, then read the format off the file it generated. The failures all live on the other path: adding a file to a project that already exists, where there is nothing to scaffold from.Fix 1 — global flags in either position
Refused outright; the command does not run. One agent hit this three times in one session —
--json,-v,--quiet— and named flag position first when asked what it most wished the tool had said up front.This was already known and already documented. The documentation is what
tools/verify_release.pywas written against when it reported two false failures. Documenting a trap does not remove it, so all 24 subcommands now take the flags themselves.argparse.SUPPRESSis load-bearing: without it the subparser default overwrites the global value, soplainsong --json info xwould parse and then silently lose its--json— a loud failure traded for a quiet one. A subcommand defining its own flag keeps it, soinfo --verboseis untouched.Fix 2 — info-level diagnostics were displayed by nothing
_diagnosticsfiltered to errors and warnings, so a whole severity was write-only: produced by the parser, shown by no command — includinginfo --verbose, which documents itself as showing every diagnostic available.Title: My Songis the natural way to name a piece and is not the notation:Title dropped, stray row silently promoted to a section,
--verbosesilent. The library agent hit this and had to readparser.pyto find out why. Default output is unchanged; the corpus emits no info-level diagnostics, so nothing gets noisier.Fix 3 —
--helpshows the notationEvery cold agent ran
--helpfirst, and one asked for precisely this: show the shape so it need not generate a file to learn it. Six lines now do.That makes
--helpa claim about the language, so it is held to the bar this repo holds prose to —TestTheHelpTeachesRealNotationcompiles the sample, requires it to sound, and requires zero diagnostics. PuttingTitle:in the help fails two of the three.What I withdrew, and what I can't claim
I initially called a missing lyric-binding diagnostic a third instance of the recurring two-sources bug. It isn't —
cmd_lyricsdeliberately arranges twice, and that warning belongs to bound mode, which isn't the default. The agent's reading was right and mine was wrong. No change made.And the two follow-up probes did not exercise fixes 1 or 2. Both took the scaffold path and both used the old global flag position. They are evidence of no regression and nothing more; the fixes rest on direct before/after plus mutation tests.
Verified
Every guard mutation-tested, because a guard that cannot fail is decoration:
--verboseto error/warning onlyTitle:in the help sampleverify_release.py --stage wheelruff check/format --checkcheckover 6,337 sourcesDocs corrected in the three places that taught the old constraint:
CLAUDE.md,docs/integration.md(both passages), andAGENTS.md, whose "Every command takes--json. Use it." is the sentence that caused the original bug.