Repository navigation
Add a dedicated page on quantitative assessment for format selection #207
Description
Context
PR #205 adds the format selection decision framework. It contains a short optional section, "Optional: make it quantitative". That section names the benchmark harness and gives two warnings. It does not explain how to build or how to read a score.
What the page must cover
The page must show a reader how to go from measurements to a ranked short list.
The model. A weighted mean of criteria. Each criterion gives a number from 0 to 1.
Qualitative criteria, which need no benchmark run:
- use-case fit: the intent mix, applied to a per-format suitability matrix
- ecosystem maturity: the L1 to L5 level, mapped to a utility
- interpretation encoding: the encoding level, mapped to a utility
Measured criteria, which need a benchmark run:
- size: the output volume against the source
- performance: the measured read, write, and display, blended by the intent mix
How to read the result. The rank, and the margin to the second candidate. A small margin means that a benchmark run, or a review of the weights, decides the choice. There are no verdicts and no probability readout.
The suitability matrix as a shared expert reference. It is useful on its own, with no benchmark run.
Two rules that are easy to get wrong:
- Object size and tier fit are reported information. They are not a scored penalty. A storage policy is a property of your infrastructure, not of the format.
- The intent mix and the criterion weights are the reader's input. They are not properties of the data.
The sensitivity sweep. A single score proves nothing. A sweep that holds the ranking across a range of weights is the useful artifact.
What is needed first
The page describes a method that a reader must be able to run. Today the scoring layer is not available in cng-formats-benchmark in a documented and reproducible form. The harness measures the metrics. It does not yet publish the suitability matrix, the default weights, or the sensitivity sweep.
Integration work comes first:
- Expose the scoring layer in the harness, with the criteria and the default weights
- Publish the suitability matrix, and record the source of each value
- Add a sensitivity sweep to the harness output
- Document all three in the harness documentation
The guide must not describe a model that a reader cannot reproduce.
Acceptance criteria
- A new page describes the method, and it links to the harness for every number.
- The short section in the decision framework links to the new page.
- A reader can reproduce a ranking on their own data with the harness.
Activity