Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions corpus/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,13 @@ Each of these is a question the format would otherwise leave to whoever writes t

**Empty directories cannot be expressed**, because git does not store them. A case needing one is a reason to extend the format.

**A symlink is compared as a link, not as its target.** `tree/` is copied with links preserved, and a symlink in `expected/` matches only a symlink whose target string is the same. This has to be stated rather than left to a reader, because the obvious implementation reads through the link and then accepts a run that replaced the link with a regular file holding the target's bytes — which is the one rewrite a symlink case exists to rule out, and the harness here comments the same reasoning at the snapshot it takes.

**The working directory is the copied tree.** `argv` therefore holds relative paths and needs no substitution, and a case may put a `.unwrapignore` in its `tree/` and have it found.

**stdout is compared byte for byte. stderr is not compared at all.** Byte-identical error prose across two languages is maintenance cost with no user-visible payoff, so the tier asserts the first and leaves the second free. Stating the boundary is the point; an unstated one gets litigated at every divergence.

**No case asserts `--help` or a usage message.** Those carry the program name, and the program name is exactly what differs between implementations — `unwrap-markdown-prose` against `unwrap-markdown-prose-rs`, and `__main__.py` when the harness invokes a module. Usage goes to stdout on `--help` and to stderr on a parse error; the first can never match and the second is not compared.
**No case asserts `--help` or a usage message.** Those carry the program name, and the program name is exactly what differs between implementations — `unwrap-markdown-prose-py` against `unwrap-markdown-prose-rs`, and `__main__.py` when the harness invokes a module. Usage goes to stdout on `--help`, which can never match. A parse error writes its diagnostic to stderr, which is not compared, and whether usage accompanies it is itself an implementation difference: the Python prints both and the Rust prints the diagnostic alone.

**Line endings are bytes.** `.gitattributes` marks `corpus/cli/**` as `-text -diff` for the reason the transform tier is marked: a case pinning CRLF stops pinning anything the moment git normalizes it on checkout. The net is the whole subtree rather than `*.md`, because `case.txt` and `stdout.txt` are asserted just as literally as the fixtures.

Expand All @@ -72,4 +74,4 @@ The third is the one that catches a tool writing a file it should not have, whic

Write the files and it is picked up automatically; nothing registers cases by name. Prefer a case that pins one decision, and put the argument in `why` rather than in the slug — the slug becomes the test id, and the `why` is what the next person needs when they are staring at a failure and deciding whether the rule or the case is wrong.

Generate `expected/` and `stdout.txt` by running the tool rather than by writing them out. An answer key written by hand pins what its author believed, which is the one thing a conformance case must not do.
Generate `expected/` and `stdout.txt` with the harness rather than writing them yourself: `REGENERATE_CLI_CORPUS=1 uv run python -m pytest tests/test_cli_corpus.py -k <slug>` rewrites both from what the reference run did, leaving `exit_code` as the one expectation you state rather than observe. An answer key written by hand pins what its author expected rather than what the tool does. Running the tool outside the harness records what the tool does, but under a different setup. The harness copies the tree, applies the modes, runs the tool inside the copy and snapshots the result; regeneration reuses all four, so a key produced any other way can be one that no run reproduces.
Loading