Skip to content

Make the CLI work the way an agent expects it to - #18

Merged
SuperInstance merged 2 commits into
masterfrom
claude/init-1wgqn0
Aug 24, 2026
Merged

Make the CLI work the way an agent expects it to#18
SuperInstance merged 2 commits into
masterfrom
claude/init-1wgqn0

Conversation

@SuperInstance

@SuperInstance SuperInstance commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Agents are most of who uses this tool, so rather than reason about what they find hard, I measured it: six Haiku-class agents were pointed at the CLI cold, with CLAUDE.md, AGENTS.md and docs/ explicitly withheld, and asked to do ordinary things — extract structured data, write a song, diagnose a broken file, drive the library.

What the tool already does well

Two probes cleared it outright, and that's worth as much as the defects:

  • The library API needed no work. The agent found compile_text on its first read of __init__.py, made zero wrong guesses, hit no AttributeError or TypeError. Asked whether there was an obvious front door: "Absolutely yes."
  • The diagnostics needed no work. On a file with six planted faults — bad key, bad tempo, bad metre, unreadable chord, short melody, unbindable lyrics — the tool named all six explicitly. Nothing had to be inferred from the music sounding wrong.

The split that explains everything else

Path taken Outcome
Reached for plainsong new, or an existing file Sailed through
Hand-authored blind Flag position 3×; Title:

plainsong new is the strongest affordance here — one agent reached for it even after being told not to, then read the format off the file it generated. The failures all live on the other path: adding a file to a project that already exists, where there is nothing to scaffold from.

Fix 1 — global flags in either position

$ plainsong info song.song --json
plainsong: error: unrecognized arguments: --json

Refused outright; the command does not run. One agent hit this three times in one session--json, -v, --quiet — and named flag position first when asked what it most wished the tool had said up front.

This was already known and already documented. The documentation is what tools/verify_release.py was written against when it reported two false failures. Documenting a trap does not remove it, so all 24 subcommands now take the flags themselves.

argparse.SUPPRESS is load-bearing: without it the subparser default overwrites the global value, so plainsong --json info x would parse and then silently lose its --json — a loud failure traded for a quiet one. A subcommand defining its own flag keeps it, so info --verbose is untouched.

Fix 2 — info-level diagnostics were displayed by nothing

_diagnostics filtered to errors and warnings, so a whole severity was write-only: produced by the parser, shown by no command — including info --verbose, which documents itself as showing every diagnostic available.

Title: My Song is the natural way to name a piece and is not the notation:

$ plainsong info titled.song
(untitled)  --  Dm, 96 bpm, 4/4
2 sections, dialect: absolute       ← the file has one section

Title dropped, stray row silently promoted to a section, --verbose silent. The library agent hit this and had to read parser.py to find out why. Default output is unchanged; the corpus emits no info-level diagnostics, so nothing gets noisier.

Fix 3 — --help shows the notation

Every cold agent ran --help first, and one asked for precisely this: show the shape so it need not generate a file to learn it. Six lines now do.

That makes --help a claim about the language, so it is held to the bar this repo holds prose to — TestTheHelpTeachesRealNotation compiles the sample, requires it to sound, and requires zero diagnostics. Putting Title: in the help fails two of the three.

What I withdrew, and what I can't claim

I initially called a missing lyric-binding diagnostic a third instance of the recurring two-sources bug. It isn'tcmd_lyrics deliberately arranges twice, and that warning belongs to bound mode, which isn't the default. The agent's reading was right and mine was wrong. No change made.

And the two follow-up probes did not exercise fixes 1 or 2. Both took the scaffold path and both used the old global flag position. They are evidence of no regression and nothing more; the fixes rest on direct before/after plus mutation tests.

Verified

Every guard mutation-tested, because a guard that cannot fail is decoration:

Mutation Result
Don't attach flags to subparsers 24 assertions fail
Revert --verbose to error/warning only fails
Put Title: in the help sample 2 of 3 fail
stdlib-only (the 12 platform jobs) 705 tests, 0 errors, 0 failures
pytest job clean
specs 7/7
verify_release.py --stage wheel 10 passed, 0 failed
ruff check / format --check clean
corpus fingerprint 6,321 files compile to identical music
check over 6,337 sources 4 warnings — the same 4 master reports

Docs corrected in the three places that taught the old constraint: CLAUDE.md, docs/integration.md (both passages), and AGENTS.md, whose "Every command takes --json. Use it." is the sentence that caused the original bug.

claude added 2 commits August 24, 2026 21:42
Four Haiku-class agents were pointed at this tool cold, with CLAUDE.md, AGENTS.md
and docs/ withheld, and asked to do ordinary things: extract structured data,
write a song, diagnose a broken file, drive the library. The point was to find
out what the tool is like to use without alignment, since agents are most of who
uses it. Two findings were worth acting on.

Global flags only worked before the subcommand. `plainsong info song.song --json`
was refused outright, and the refusal aborts the command. One agent hit it three
times in one session -- --json, -v and --quiet -- and named flag position first
when asked what it wished the tool had told it up front. This was already known
and already documented; the documentation is what tools/verify_release.py was
written against when it reported two false failures. Documenting a trap does not
remove it, so all 24 subcommands now take the flags themselves.

argparse.SUPPRESS is the load-bearing part: without it a subparser default
overwrites the global value, so `plainsong --json info x` would parse and then
quietly lose its --json. A loud failure traded for a silent one is the worse
trade. `info` keeps its own --verbose rather than having it redefined.

And info-level diagnostics were displayed by nothing at all -- not compile, not
check, not `info --verbose`, which documents itself as showing every diagnostic.
A whole severity was write-only. That is why `Title: My Song`, the most natural
way to name a piece and not the notation, dropped the title silently, turned the
row into a phantom section, and left `(untitled)` and a section count one too
high as the only evidence. A fourth agent hit it and had to read parser.py to
find out why. --verbose now includes them, in their own voice rather than
painted as warnings; default output is unchanged and the corpus emits none.

One thing the probes cleared rather than condemned: the library API. An agent
found compile_text on its first read of __init__.py with no wrong turns, and the
diagnostics on a deliberately broken file named all six faults explicitly. Those
needed no work, which is worth recording as much as the defects.

Both guards were mutation-tested: removing the subparser flags fails 24 of the
new assertions, and reverting the verbose filter fails the other. Verified in
each CI environment -- stdlib-only 702 tests 0 errors, pytest clean, specs 7/7,
wheel 10/10, ruff clean, corpus fingerprint unmoved at 6,321 files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PBAjxy7cD6DzJ72NX8TJEc
Six agents were pointed at this CLI cold. The split was clean: every one that
reached `plainsong new` succeeded, and the two that hand-authored blind hit the
traps. One reached for `new` even after being told not to, then read the format
off the file it generated -- which is the strongest evidence available that
scaffolding is the path agents actually take, and that it works.

The failures were on the other path: adding a file to a project that already
exists, where there is nothing to scaffold from. There the format has to be
guessed, and the guess was `Title:`, which is not the notation and which cost a
dropped title and a phantom section. One agent asked for exactly this fix --
that `--help` show the shape so it need not generate a file to learn it.

So `--help` carries the notation now. That makes it a claim about the language
rather than decoration, so it is held to the bar this repository holds prose to:
the sample must compile, must sound, and must produce no diagnostics. Replacing
`**TRACK:**` with `Title:` in it fails two of the three guards.

Worth recording honestly: neither follow-up probe exercised the two fixes in the
previous commit. Both took the scaffold path and both used the old global flag
position, so they are evidence of no regression and nothing more. Those fixes
rest on the direct before/after and their mutation tests, not on these runs.

705 tests, specs 7/7, ruff clean, corpus fingerprint unmoved, and `check` over
all 6,337 sources reports the same 4 warnings it reports on master.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PBAjxy7cD6DzJ72NX8TJEc
@SuperInstance
SuperInstance merged commit dfc9024 into master Aug 24, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants