byod.md

Bring your own data

MicroAnt's published numbers are answers about MicroAnt's pack. Your question is different: what does Q4 do to my task? Nobody else can answer that, because nobody else has your data — and your data never leaves this machine anyway, which is the natural shape of an on-device tool.

This is the walkthrough. A complete worked pack ships with the repo at examples/support-triage/: 24 support-ticket cases, in a domain deliberately unlike the official pack's. It loads and scores in CI on every commit, so it cannot quietly rot into an example that no longer works.

1. Scaffold

microant pack init packs/my-task --name my-task

You get pack.yaml (your output shapes and tool signatures), cases.jsonl (four template cases), and a README.md. The template is opinionated on purpose: the two mistakes below are the ones that make a ruler lie, and both of them pass every other check silently.

2. Write cases

A case is a prompt, an expected answer, and the rules for what counts as right. That last part is what makes a pack a pack rather than a dataset — a dataset doesn't carry its own grading rules, and the grading rules are where the honesty lives.

{"id": "tri-02", "task": "json_schema", "split": "dev",
 "turns": [{"role": "user", "content": "omar.haddad@northwind.co.uk says he was charged twice..."}],
 "schema_ref": "triage",
 "expected": {"fields": {"priority": "P2", "category": "billing",
                         "customer_email": "omar.haddad@northwind.co.uk"}},
 "rules": {"priority": {"match": "equiv", "equiv": [["P1", "urgent", "critical"], ...]},
           "category": {"match": "exact"},
           "customer_email": {"match": "exact"}}}

Three task types: json_schema (produce a structured object), extraction (pull it out of a blob of text), function_call (choose a tool and fill its arguments).

3. Choose the match rules — by field type, never per case

rule	for	behaviour
exact	emails, ids, enums	character-exact
number, iso_minute	amounts, timestamps	normalized, then exact
set	unordered collections	order-insensitive
equiv	declared synonyms	your own list (P1 = urgent = critical)
text	names, titles, places	casefold + collapse whitespace, then exact — word order still matters
semantic	paraphrasable free text	Jaccard overlap of content tokens above a threshold

The rule kinds are a closed set. An unknown kind fails at load time — silently degrading it to exact was a real bug here once.

Two mistakes that will bite you, and neither one will warn you:

A structured field on a fuzzy rule waves through the exact damage you are hunting. In the example pack, tri-02's gold email is omar.haddad@northwind.co.uk. A quantized model that emits omer.haddad@ — one character — has hallucinated a real customer into a different one. Under exact that is a failure, which is correct. Under text or semantic it passes, and your report will tell you the quantization was harmless. There is a test in this repo (test_a_hallucinated_character_in_an_email_actually_fails) that exists solely to keep that promise honest.

A natural-language field on bare exact fails a correct answer over a comma. Ada Okonkwo vs ada okonkwo is not a model error. Use text.

And a rule belongs to a field TYPE, not to a case. If email is exact in one case it is exact in every case. Choosing leniently where your prompt happened to be vague and strictly where it was precise measures how you worded your prompts, not how the model behaved — and every case still loads, every number still renders, and nothing anywhere goes red.

semantic deserves one more warning, because it is the rule that looks smartest and is the dumbest: it is lexical overlap, not meaning. Against a short gold string it will accept a single substituted word — swapping winner for loser scores 0.714, identical to a genuine paraphrase, so no threshold separates them. Read a semantic pass as "the words that had to appear did", never as "the model understood".

Which is why the example pack uses five of the six rules and never once reaches for semantic. Every field in a support ticket is either structured (an email, a ticket id, an amount, a timestamp, an enum) or a name — and a name is text. That is not an oversight; it is the answer to the question you should ask before typing semantic: is this field genuinely paraphrasable, or did I just not want to think about what "right" means? Most fields are the second one.

4. Size the pack before you author it, not after

This is the step everyone skips, and it is the one that decides whether the week you are about to spend produces a number or a shrug.

Bootstrap confidence intervals over a small case set come out roughly half a scale wide. MicroAnt will print them honestly, and honestly they will say nothing. Measured against real data (power-analysis.md):

the damage you want to detect	cases you need
0.15	~48
0.10	~68
0.05	~140

And that is the optimistic column — it assumes no reverse flips, cases the quantized model gets right and the 16-bit baseline gets wrong. Reverse flips do not cancel out; they leave the mean where it was and widen the interval. At a 10% flip rate, detecting a 0.05 effect needs more than 400 cases.

The example pack has 24 cases and therefore cannot detect a 0.10 drop. It is a tutorial, not a ruler, and it says so in its own pack.yaml. Everything MicroAnt prints about it will be honestly, uselessly wide. Seeing that is part of the lesson: it is exactly what your first pack will look like if you don't do this arithmetic first.

5. Split dev / test — and understand what you are spending

Every case declares split: dev or split: test.

• prescribe selects a scaffold rung on the dev half.
• confirm reports that rung's number on the test half — cases it never saw.

A number quoted from the same cases a choice was made on is optimistic by however much the choosing had to reach. That is not a technicality; it is the difference between "this scaffold works" and "this scaffold worked on the cases I tuned it against."

confirm requires a --purpose and writes to a test-set ledger. Held-out power erodes with every look. The ledger does not stop you looking; it makes the erosion auditable instead of invisible.

A case with no split defaults to dev, so a pack that never thought about this lands entirely on the selection side and never fabricates a held-out number.

6. Run it

microant family qwen3
# choose one concrete member from the family list, then run it:
microant profile   --model qwen3-1.7b --backends mlx --tiers /4bit,bf16 --pack examples/support-triage
microant prescribe --model qwen3-1.7b --backends mlx --tiers /4bit,bf16 --pack examples/support-triage --threshold 0.9
microant confirm   --model qwen3-1.7b --pack examples/support-triage --purpose "first look"

--tiers is <gguf tiers>/<mlx tiers>, so /4bit,bf16 means "no GGUF tiers, two MLX ones".

MicroAnt is family-first at the selection layer: the user starts from a family (for example qwen3 or gemma4), then chooses one concrete member of that family to run. The report is still produced per concrete member, because the actual ladders / baselines / tiers differ by artifact.

• profile — what quantization costs you. Where cases die (parse_ok → schema_valid → content_correct), damage against the chain's own 16-bit baseline, with a bootstrap CI on every number and a s/case cost column beside it.
• prescribe — the lowest scaffold rung that clears your bar: L0 bare, L1 grammar-constrained decoding, L2 few-shot, L3 self-check retry. "L1 is enough, don't write the retry logic" is a prescription too. It also draws the dual-ring radar: bare model inside, scaffolded outside, and the area between is what engineering can buy back.
• confirm — re-measures only the chosen rung on the held-out half. It is allowed to say the prescription did not hold, and it will.

7. Read the report like a skeptic

Three things the report does that you should not smooth over:

inconclusive is not no. A CI that straddles zero means underpowered, not harmless. An underpowered comparison and a genuine null result look identical from the outside, and only the case count tells them apart. Go back to step 4.

Every table is rendered twice, under a strict and a lenient parser. A conclusion that flips when the parser gets more forgiving was never about the model — it was about the parser.

Numbers do not transfer across kernel paths. Same GGUF file, same greedy decoding, temperature 0: llama.cpp's CPU and Metal kernels produce identical output on only 2 of 10 prompts (the spike). Your numbers describe (runtime, kernel backend, quantization file, sampling config). A Mac running Metal and a phone running NEON will not agree, and no amount of seed-fixing changes that.