Skip to content
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions corpus/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,13 @@ Each of these is a question the format would otherwise leave to whoever writes t

**Empty directories cannot be expressed**, because git does not store them. A case needing one is a reason to extend the format.

**A symlink is compared as a link, not as its target.** `tree/` is copied with links preserved, and a symlink in `expected/` matches only a symlink whose target string is the same. This has to be stated rather than left to a reader, because the obvious implementation reads through the link and then accepts a run that replaced the link with a regular file holding the target's bytes — which is the one rewrite a symlink case exists to rule out, and the harness here comments the same reasoning at the snapshot it takes.

**The working directory is the copied tree.** `argv` therefore holds relative paths and needs no substitution, and a case may put a `.unwrapignore` in its `tree/` and have it found.

**stdout is compared byte for byte. stderr is not compared at all.** Byte-identical error prose across two languages is maintenance cost with no user-visible payoff, so the tier asserts the first and leaves the second free. Stating the boundary is the point; an unstated one gets litigated at every divergence.

**No case asserts `--help` or a usage message.** Those carry the program name, and the program name is exactly what differs between implementations — `unwrap-markdown-prose` against `unwrap-markdown-prose-rs`, and `__main__.py` when the harness invokes a module. Usage goes to stdout on `--help` and to stderr on a parse error; the first can never match and the second is not compared.
**No case asserts `--help` or a usage message.** Those carry the program name, and the program name is exactly what differs between implementations — `unwrap-markdown-prose-py` against `unwrap-markdown-prose-rs`, and `__main__.py` when the harness invokes a module. Usage goes to stdout on `--help`, which can never match. A parse error writes its diagnostic to stderr, which is not compared, and whether usage accompanies it is itself an implementation difference: the Python prints both and the Rust prints the diagnostic alone.

**Line endings are bytes.** `.gitattributes` marks `corpus/cli/**` as `-text -diff` for the reason the transform tier is marked: a case pinning CRLF stops pinning anything the moment git normalizes it on checkout. The net is the whole subtree rather than `*.md`, because `case.txt` and `stdout.txt` are asserted just as literally as the fixtures.

Expand All @@ -72,4 +74,4 @@ The third is the one that catches a tool writing a file it should not have, whic

Write the files and it is picked up automatically; nothing registers cases by name. Prefer a case that pins one decision, and put the argument in `why` rather than in the slug — the slug becomes the test id, and the `why` is what the next person needs when they are staring at a failure and deciding whether the rule or the case is wrong.

Generate `expected/` and `stdout.txt` by running the tool rather than by writing them out. An answer key written by hand pins what its author believed, which is the one thing a conformance case must not do.
Generate `expected/` and `stdout.txt` with the harness rather than by writing them out: `REGENERATE_CLI_CORPUS=1 uv run python -m pytest tests/test_cli_corpus.py -k <slug>` rewrites both from what the reference run did, leaving `exit_code` as the one expectation you state rather than observe. An answer key written by hand pins what its author believed, which is the one thing a conformance case must not do, and running the tool yourself instead of through the harness reintroduces the same risk one step further back -- the harness owns the copy, the modes, the working directory and the snapshot, and a key produced beside it can disagree with the one it would compare.
Comment thread
Copilot marked this conversation as resolved.
Outdated
Loading