Release 0.2.0: the analysis chain, and the browser workflow that feeds it - #17
Merged
Merged
Conversation
A template's verifier field now holds either one verifier or an array of declarative verifiers, all of which must pass. Every element runs, and the verdict carries a per-element report so a failure names the element rather than collapsing to a boolean. The array references the declarative union in the schema, so an executable element is a load-time violation rather than a runtime refusal, and lists do not nest, so the depth limit and the eager pre-walk both go away. Spec version bumped to redrob-verifiable-task/v2. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Spec section 6.2 now describes the field shape, the per-element report and the structural rejection of executable elements; section 11.1 records the v2 migration. Decision record 0003 covers what was decided unreviewed. Four negative controls added: the element report comparison catching a short circuit that case verdicts do not, the same short circuit hiding an executable element behind a failing one, the schema rejecting an executable element, and a schema-rejection row proving its valid counterpart. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
main had been the only long-lived branch, which is how the Generate module came to be squash merged straight into what is meant to be deployable. - develop is created from main and is now the base for all ordinary work - CONTRIBUTING documents the model with a diagram, the branch prefixes, worked commands for cutting a release and making a hotfix, and the three QA layers -- noting that CI has no browser, no provider key and no GPU, so the UI, a live call, /deploy and upgrade-in-place are covered by nothing until someone tests them on a release branch - the merge strategy is stated: squash into develop, real merge commits for release and hotfix in both directions, because a squashed hotfix reaches main and develop as two unrelated commits and the next release merge then conflicts over a fix that was applied deliberately - both workflows build develop on push; master is dropped from the triggers because no such branch exists Decision record 0004 covers the naming, why the Generate merge on main is not reverted, and the repository settings that still have to be set by hand for any of this to be enforced rather than requested. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Part C of the Generate work: all_of is replaced by a verifier field that accepts either a single verifier object or an array, with a normative per-element report and executable elements refused by the schema rather than at runtime. Spec bumped to v2. Resolves a README conflict where both branches extended the decision record list, and drops a duplicate Contributing bullet the branching change had introduced. Known to still need fixes; it lands on develop rather than main so the follow-ups are small branches off an integrated base instead of a long-lived branch drifting from it. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
The href was updated to verifiable-task-v2.md but the link text still read v1, so the README announced the wrong spec version on the line that introduces it. Found while resolving the merge; the file it points at was already correct, so no link checker would have caught it. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Generate can now run one study end to end: redrob-generate study takes a config and writes a validated results artifact plus an aggregate table. - Verdict records carry the producing implementation, its version and its Unicode table version. Python is authoritative, TypeScript is display only. The two runtimes really do read different Unicode tables here (15.0.0 against 16.0), which is why the distinction is provenance rather than a correctness claim. - Locale layers carry translation_status, and hi, hi-Latn and ko stubs render the English text verbatim so the pipeline can be exercised without fabricating translations. - Two new fields: fertility_tokenizer on a study config, code_mix_ratio on an instance. The latter is always null; defining the measurement is a human decision. - Publication refuses an untranslated locale and refuses a verdict this project did not produce. - The locale pattern gained a script subtag so Hinglish is expressible as hi-Latn. Model invocation goes through the harness callModel via a thin bridge, so there is no second model client. The mock provider needs no network. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
- spec: two sentences on Python being normative for publication, plus the translation_status and code_mix_ratio fields and the script subtag. - STUDY.md: config format, worked example, and a plain statement that the stub locales cannot be published. - Decision record 0005 records the five choices taken without review, the largest being that the bundled tokenizers are offline proxies rather than model tokenizers. - Six new negative controls, one per new invariance claim. All 17 detect their break. - Three new DoD steps: reproducible study, artifact shape, publication gates. All 16 steps pass. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
SUMMARY-4 records what was built, the five decisions taken without review, the deviations and their cost, and the command output behind each definition-of-done item, plus the list of things left alone for being out of scope. The logs/ ignore rule exists because a build transcript is reproducible by re-running the build. A written report is neither reproducible nor a transcript, so the rule was catching it by accident, and three passes of reasoning have been sitting untracked as a result. The pattern becomes logs/* with the reports negated -- git will not re-include a file whose parent directory is excluded -- and the earlier three are backfilled in the same commit so the record is not left half present. Per-step .log files stay ignored; CI uploads those as workflow artifacts. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
…lowed There is no prettier config at the root, so running the binary bare formats to its own defaults -- double quotes and an 80 column wrap -- which disagrees with every file in the repository. A previous step in this pass ran it over six files and committed the result, which churned reader.ts and index.ts far beyond the lines actually being changed and made the real edit hard to find in a diff. reader.ts and index.ts go back to their committed formatting with the semantic edits kept. The four files written during this pass are converted to the repository's single-quote style so they match their neighbours. No behaviour changes: tsc is clean and all 665 TypeScript tests pass. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Three additions behind a Generate UI, all in the generate subpath so nothing reaches the main harness barrel or a client bundle. catalog.ts reads the template tree and the shipped study configs from disk: which families exist, which locales each has, and what review status each locale claims. It deliberately needs no Python. Sampling does -- this implementation reads and verifies but does not generate -- so a catalog that also needed the interpreter would leave a workbench without one showing an empty page instead of a list of what exists and a note about what is missing. A locale layer that declares no translation status reads as untranslated rather than as anything friendlier. Absence of a claim is not a review, and the failure mode of defaulting upward is silent: a green chip, a plausible table, and a reader believing a number describes Korean. python-client gains emitWithPython and studyWithPython alongside the existing verify, reusing the same spawn and the same treatment of absence as an expected state. studyWithPython reads exit 3 as a result rather than a failure: the CLI writes the artifact before the publication gate runs, so a refusal still has an artifact to show and a reason worth reading. 13 new tests, one of which asserts every stub locale still reports the same prompt length as its English source. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Generate has had a spec, a generator, two verifier implementations and a study runner, and no way to reach any of it without a terminal. This adds /generate to the module nav next to Compare, Evolve and Deploy. Two stages. Templates lists every family, the locales it has and how reviewed each one is, and samples instances on demand so you can read the rendered prompt, the parameters behind it and the verifier that will score it. Study runs a shipped config and renders the aggregate tables together with the provenance block and the publication verdict. Three deliberate choices in the routes: The publication gate always runs. The CLI writes the artifact before checking it, so asking costs nothing, and reporting publishable without having checked would be the page asserting something it never established. The refusal shown is Python's own rather than a second copy of the rule living in the browser. A study config that reaches a provider is refused by the route, not merely hidden in the UI. Real model calls stay on the command line where the spend is a decision someone typed. Paths from the browser are confined rather than trusted: a preview path must resolve inside templates/, and a study config must be one the catalog offered. Otherwise these would be arbitrary-read and arbitrary-execute primitives with a JSON body. Absent Python is reported once, up front, with the install command, and the catalog still renders because it is read from disk. The alternative is discovering it one dead button at a time. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
The README said Generate had no page, no route and no navigation entry, which is no longer true. It now appears in the module table and the paragraph about what exists says so, including that the page still works without Python. The locale note was also understating things: it said templates are English-only and the locale structure merely exists, which reads as though hi, hi-Latn and ko are absent rather than present and holding English. STUDY.md gains a section on the page, covering the three things it does deliberately -- always asking the gate, refusing to spend, and saying when it cannot help. Negative control 18 defaults an undeclared translation status to native-reviewed and confirms the catalog tests catch it. This is the failure mode worth a control because it is silent and it is upward: nothing errors, the chip turns green, and a reader believes a number is about a language nobody translated. The definition-of-done build check now expects generate.html alongside the other three module pages. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Manual testing turned up four things worth fixing. The right-hand panel was an empty white void until you clicked something. The page is reached to look at templates, so the first family is now selected and sampled on load and there is nothing to click before seeing a prompt. The locale chips in the template list read 'single reviewer, untranslated, untranslated, untranslated' -- which says a family has stubs but not which ones, and which one is which is the only part a reader can act on. They now carry the locale tag and keep the colour. A chip inside a column flex container stretched to the full width of its card, so 'mock only, no spend' was drawn as a long pill across the study config. Three labels were terse enough to be ambiguous: 'instances' next to a number field, '#0' on an instance header, and 'Expected' over the verifier config. Provenance now also says in one line why two runtimes are listed at all. The loading flag is gone rather than fixed. Results are keyed by the request that produced them, so switching template or locale shows nothing instead of briefly showing the previous template's instances, and 'loading' follows from having neither a result nor an error for the current key. One less piece of state to keep in sync, and it satisfies the lint rule against setting state synchronously in an effect without an escape hatch. Verified in a browser: switching templates never shows stale instances, the count field does not re-sample per keystroke, and switching locale re-samples to identical seeds -- seeds carry no locale term, which is what makes the paired deltas paired. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
The stylesheet named 123 colours directly and the charts named 18 more. A literal in a rule is a colour that cannot follow a theme, so a second palette was impossible before this: nothing to switch. Every literal below the new token layer is now a token, and the layer defines two palettes under one set of names. The rules that needed judgement rather than a table: - White is three different roles. A panel background, a label on a saturated fill, and titlebar hover text all wrote #fff, and they diverge in dark mode -- the panel darkens, the label does not. They are --panel, --on-solid and --titlebar-ink-max. - Status tints stay explicit per theme. Mixing them from the base colour is tidier and was tried, but a mix of --warn into white lands on beige where the light theme has always used cream: the elegant mechanism quietly restyles the shipped product. - The titlebar and the terminal are dark in both themes, so their tokens are named for the surface they sit on rather than for the theme. - Charts get var() in their SVG attributes, which resolve as CSS and so re-colour on a theme change with no re-render. xterm paints to a canvas and cannot, so it reads the same two tokens through getComputedStyle -- which is also what stops the canvas and the frame around it from drifting apart. No visible change yet: the light palette holds the previous values. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Three states rather than two: light, dark, and follow the OS. The preference and the resolved theme are kept apart so the switch still reads System after the OS flips, instead of silently rewriting the choice to whatever the OS happened to be. Applied by an inline script in <head>, which is the only place it can be applied. A dark-mode user whose page paints white and corrects itself is the most visible defect a theme can have, and no effect can prevent it, because effects run after paint. Because that script always resolves to a concrete light or dark on <html>, the stylesheet needs one [data-theme='dark'] block rather than that block plus a duplicate inside a prefers-color-scheme query, and the two cannot drift apart. The choice lives in localStorage and in the OS setting, neither of which React owns, so it is read with useSyncExternalStore rather than mirrored into state. Mirroring would mean writing state from an effect on mount -- a cascading render, and a second copy of the truth that goes stale when another tab changes it. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Settings gets the three-way control, in its own card above the saved settings and outside the save flow -- this preference lives in the browser rather than in .env and applies on click, so a Save button next to it would be a lie. Three radios rather than a toggle: follow the OS is a distinct answer from either fixed value, and a checkbox cannot express it. The titlebar gets a single button that cycles the same three states, because that bar is 48px tall and already shared with every module's own controls; it is the shortcut for someone who has just walked into a dark room, not the place to explain the options. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
A dark palette is easy to get subtly wrong in a way that looks fine to whoever picked the colours and is unreadable to everyone else. A review of the first draft reported no defects; these numbers found three, one of them severe: - The dark titlebar sat at 1.02:1 against the page behind it. Not subtle -- invisible. In light mode the bar is the dark element and needs no help separating itself; in dark mode nothing is, so it now separates by hue and by sitting slightly lighter. - Seven controls filled with --brand under a white label. --brand is lifted in dark mode so it stays readable as text on a dark panel, which makes it far too light to sit under white. Fills are now their own tokens, kept saturated. - The infeasible dots on the Pareto chart were at 1.78:1 against the panel. The bar for dark is the bar light already clears. An absolute WCAG floor would fail pairings light has always failed -- hairline borders, muted grey on the page -- and the only way to pass would be to restyle the light theme, which is not what adding a dark mode is for. Surface joins are judged on being perceptible rather than on 3:1, which governs controls and meaningful graphics; the titlebar's own contents are checked as text and clear 4.5:1. The tokens are read out of globals.css rather than restated, because a copy of the palette in a test is a copy that goes stale and then passes while the app is broken. A companion assertion fails the build if any colour below the token layer stops being a token, which is how the palette stays the only place colour is decided. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Resolving every declaration in the old stylesheet and the new one and diffing the results turned up two rules where an opaque token had replaced a translucent value: the browse badge's border and the completed-step marker. Both sit on surfaces that are not the panel, so flattening them paints over what is behind. They keep their alpha now. Eleven differences remain and are deliberate: near-duplicate values collapsed onto one token, none of them more than 3/255 in a channel or 5% alpha on an already-faint tint. The theme toggle also needed a focus ring of its own. The titlebar is dark in both themes but color-scheme follows the page, so in light mode the browser was drawing its default ring dark-on-dark, on a control that had just been added. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
What matters about that script is which theme it decides on, and a regex over its source cannot tell whether the fallback chain is right. It is executed against a stub document instead, once per case: an explicit choice beating the OS, no choice or 'system' following it, a nonsense stored value, storage that throws the way private browsing does, and color-scheme being set so native controls follow. Guarding where it lives is a separate assertion, because the flash it prevents is a property of running in <head>. Measured: with the script, every frame of a dark reload is at brightness 33/255; with it removed, every frame is at 234. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
The contrast check is worth pointing at from the README because it is not obvious that 'yarn test' fails on a colour. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Every colour in the workbench becomes a theme token, and the app follows the operating system's light or dark setting unless told otherwise from the title bar or the Appearance card in Settings. Lands here rather than on develop because it themes the Generate page added on this branch as well as everything else. The theme resolves in an inline script before first paint, which is the only place it can resolve without showing a dark-mode reader a white page that corrects itself. Contrast is asserted rather than eyeballed: a review of the first draft reported no defects and measurement found three, including a title bar sitting at 1.02:1 against the page behind it. yarn test now fails if any pairing is worse in dark than in light, or if any colour below the token layer stops being a token. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Two pieces of work that share a stylesheet, so they integrate together. Generate becomes usable end to end. Python is normative for verdicts and TypeScript verdicts are marked non-authoritative, because the two read different Unicode tables and only one of them may be published from. Templates gain a locale layer with a declared translation_status, a study runner turns a config into a validated results artifact, and a /generate page makes all of it reachable without a terminal. The publication gate always runs, a config that would reach a provider is refused by the route rather than merely hidden, and paths arriving from the browser are confined. The theme turns every colour in the app into a token and adds a second palette. The workbench follows the operating system unless told otherwise from the title bar or Settings, resolved before first paint so a dark-mode reader is never shown a white page that corrects itself. Contrast is asserted rather than eyeballed, and the light theme was diffed declaration by declaration against its previous self to confirm the refactor did not restyle it. Both PRs were green on all seven checks before merging, and the sixteen checks of verify-generate-dod.sh pass on the merged tree. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Three primary buttons existed. `app-run-btn` was teal, 28px and square; `deploy-op-btn.is-primary` was brand blue, 32px and square; `settings-save` was brand blue, 36px and rounded. Teal appears nowhere else in the product except the progress bar, so the one control the eye should land on first was the only thing wearing a colour the design does not otherwise use, and a Run button four pixels shorter than the Discard beside it reads as a mistake. One geometry for all three roles, brand blue for primary, and a titlebar override so the compact bar keeps its 26px buttons. Compare's Setup action gets a panel to sit on rather than floating loose under the two cards. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
The list read `extraction.quarterly_ledger`, `format.release_note`, `math.linear_equation` — three ids in a column, scanned by their least distinctive end. The title is derived rather than authored: the spec has no title field, and adding one would put a second name on every template free to drift from the id the CLI is called with. The id stays on screen beneath the title, because it is what you type. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Sampling instances and then having no way to use them made the module a viewer. Templates now offers the set as a download, the prompts on the clipboard, the CLI command that reproduces the same sampling on disk, and a hand-off that opens Compare with the prompts already loaded as a custom set. Study offers the result artifact, which is the deliverable the tables below it are only a reading of, and says plainly that the download is the only copy. The hand-off travels through sessionStorage rather than the URL — a dozen prompts do not fit in a query string — and is read once and removed, so a later visit to Compare does not silently re-apply a set the reader has forgotten about. It carries no gold answers, because a verifier is not a reference string; the page says so and points at the preference tournament. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
It was four cards on one long scroll: the GPU host fields sat below the fold behind six provider keys, and there was no way to see what the page contained without scrolling all of it. Sections on the left, the chosen one on the right, matching Deploy. The nav carries how many values each section has set, and a dot where edits are pending, because Save applies every pane at once and that should not be something you discover by pressing it. Appearance is a section like the others even though it saves nothing. Keeping it out of the nav to reflect that implementation detail would only make it harder to find. Deploy's /settings#HF_TOKEN link now opens the pane holding the field before scrolling to it, and answers a hash change as well as a load. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
`--run-ink` existed only for text on the teal Run button, which is now brand blue; nothing sits on `--run` any more, so the token and its contrast pairing go with it. Teal stays as the progress accent, which is what it now solely is. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
A solid fill reads tighter than an outline at the same padding: the ghost's border supplies an edge the eye takes as breathing room, and blue to the edge does not. Two more pixels each side, on all three roles so the row still agrees. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Found with a live key. On GSM8K the task cap is 512 tokens; qwen3.7-flash spent all 512 thinking, returned no content, and the harness raised "Empty model response". The eval scored that 0 and the ranking reported one of the better models on the board as the worst: 10% against llama-3.1-8b's 70%. The same run now reports 100%. That is the one failure an evaluation tool may not have — a limit of our own making, published as a fact about the model. When a reply is cut off and the tokens went on reasoning the caller never asked for and cannot see, the budget is raised once and the question asked again. Ordinary models hitting the cap are left alone: that is the cap doing its job. Empty and truncated replies are also no longer retried three times. These calls go out at temperature 0, so the retry bought an identical answer — and after the escalation it would have bought it at four times the budget. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Found with a live key. Seed model was set to Llama 3.1 8B and the run manifest recorded it — `openrouter:meta-llama/llama-3.1-8b-instruct` — while every rollout went to openai/gpt-4o, because the seed candidate took modelCatalog[0] and the page sends its model-picker selection as the catalog. The report then attributed the result to a model that had not been run, and the run billed at roughly fifty times the rate asked for. The seed candidate now takes the seed model. Re-run live, the manifest and the candidate name the same model. Worth saying plainly: both halves of this were wrong in the same direction. The artifact asserted a model it had not measured, which is the failure an evaluation tool exists to prevent. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Found on a live custom-goal run. The reflector is asked for
{ lesson, instruction, ... }; the instruction is the longest field, so the
900-token cap landed inside it and the object never closed. The fallback then
used the whole raw reply as the evolved instruction, so the prompt driving
every later rollout — and the evolved prompt the reader copies out — began with
a ```json fence and a "lesson" key.
Quality still rose 46% to 62%, which is the uncomfortable part: the artifact
was visibly malformed and nothing downstream objected.
A cut-off reply now gives up its instruction and leaves its syntax behind. When
nothing can be recovered the parent instruction is kept, because raw JSON is
never a better prompt than the one we already had. The cap goes to 1800, since
a lesson plus a full replacement instruction does not fit in 900.
Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
A rubric that lists Readability (1-10), Accuracy (1-10) and three more came back as one number between 0 and 1. It could tell you the prompt scored 0.54; it could not tell you accuracy was fine and calibration was what was dragging, which is the only part anyone can act on. The dimensions are read out of the rubric the user already wrote — a bulleted or numbered line with a range in brackets — rather than asking them to restate it in a second structured field. A rubric that names none behaves exactly as before: one score, one paragraph. They travel with the batch, so reflection sees them named in its traces and can aim at the weak one, and the report shows baseline against evolved per dimension. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
The id was date_HHMMSS_dataset, unique while runs are started by one person clicking one button. Racing three models starts them in the same tick, so all three took the same id: one directory, one entry in the jobs map, three jobs writing over each other, and three event streams reporting the same run. The comparison table showed three identical rows — 60.0% to 82.0% for every model, down to the token count. That reads as suspicious rather than broken, which is the kind of result that gets believed. Four random characters after the dataset; the leading timestamp still sorts, and ids already on disk still validate. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
A single run says whether the prompt improved on one model. The question people actually have is which model to ship the evolved prompt on, and neither the strongest model nor the one that improved most is reliably the answer. Run live on a compatibility-report goal: Llama 3.1 8B gained the most, +16 points, and finished last on test at 58%. Ling-3.0-flash gained nothing at all and won at 90%, because it started at 86%. Ranking by improvement would have recommended the worst model, so the table ranks by held-out test — not by the validation split each run was fitted to, and not by the delta. Picking challengers starts the same search on each, at the same time, against the same goal, rubric and budget. A row that has not finished sorts last rather than first: an unfinished run has not earned a position. Each row carries the prompt it won with, because the answer is a pair, and showing the model without the prompt invites someone to take the name and leave the reason. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
The table needed a dozen columns — baseline, evolved, delta, test, tokens and one per rubric dimension — and the results pane is a third of the window, so everything past the model name was scrolled off. Rows read at that width and do not get wider when the pane does. The validation pair is now labelled. Beside a large test percentage, two unexplained figures that disagree read as a bug rather than as the difference between the split each run was fitted to and the one held out. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Buttons, Generate's export and hand-off, readable template names, Settings in two panes. Then four correctness bugs found by running the workbench against a real provider — a token cap scored as the model's answer, Evolve optimising a model nobody chose, GEPA shipping a JSON blob as the evolved prompt, and run ids colliding when runs start together — and the two Evolve features that came out of using it: per-dimension rubric scoring, and racing several models to see which one to ship the prompt on. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Bring PR #8 (tool-routing Compare, multi-slot Deploy, i18n) back onto develop per the release merge-back rule, keeping develop's Evolve and Generate UX under the shared i18n keys.
…rn evaluation (#9) * Rank, eliminate and re-vote on the preference ballot A group ballot could only name a winner, which says nothing about how the answers passed over compare, and the flow only ever moved forward. Ranking records every pair; eliminating records the survivors beating what went out; re-voting rebuilds one prompt from its answers and retracts its votes with an append-only undo entry, so the log still holds the change of mind. Identities now stay hidden until every prompt is decided: naming the winner of one prompt would name the model on the next. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com> * Measure and check every deploy slot from one action Install now lays down the unit and serve wrapper for every slot, so adding a slot later is a measurement rather than another install. Measure gains a whole-host run: weights for every slot download in parallel, which is where the wall time goes, then each slot is sized alone because a slot is measured against the VRAM actually free and two probes would each claim what the other is about to take. A slot that fails is reported and the rest still run. Health is genuinely parallel - independent HTTP calls to ports already serving - with output buffered per slot so it stays readable. Benchmark stays sequential: concurrent generation on one card measures contention, not throughput. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com> * Add multi-turn evaluation for text and tool conversations Single-turn scoring cannot see what conversations break on: a constraint given once and owed three turns later, a correction that has to stick, a tool result that should be read rather than fetched again. Scenarios script the user side so every model hears the same words, and each turn declares which capability it is there to test, because a bare pass rate cannot say whether the failures were forgotten instructions or bungled calls. Tools ride the same JSON contract as the single-turn routing harness rather than provider-native function calling, so a small vLLM model and a hosted API can sit the same exam; the docs say plainly what that measures and what it does not. Providers now take conversation history, which is what made any of this possible. A turn expecting a result for a call that never happened is marked desynced: the result is still delivered so the rest runs, but the turn cannot pass. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com> * Keep a half-eliminated ballot honest Knocking one answer out and then naming a winner recorded the winner as beating the answer already out, and let a model judge crown it. Both now see only what is still standing, a tie called after an elimination ties only the survivors, and the per-prompt order places the knocked-out below them, last one out highest. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com> * Report a failed weight prefetch once, with the reason Every parallel job wrote to one log and then claimed success regardless, so a failed prefetch printed both the warning and "weights ready" for the same repo. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com> * Expand the tool-routing dataset and gate its integrity The set was 100 tasks, 25 per language, split further by toolset and by call/absence until the interesting cells were single digits - too small to put a confidence interval on, which is the whole problem for a paper. It is now 324 tasks, 81 balanced per language, with absence up from 20 to 52 so BLOCK and DEFER are each measurable. A bigger set is only worth having if every task is answerable, so a validator enforces the property that matters: any argument the schema marks copied verbatim must appear in the request, while normalized values (English city, ISO date, language code) are not required to. It runs in verify:tool-routing and as a unit test, and it found a real bug on its first run - a Korean task that expected "Gangnam Station" where the request said 강남역. The registry gains more sub-10B tool callers: Qwen2.5 (Apache sizes), SmolLM2, and the xLAM function-calling specialists (eval_only, non-commercial). Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com> * Add the current sub-10B field, and a live runner to actually test it Searched the model landscape as of today rather than trusting the registry: Granite 4.1 (April 2026, dense 3B/8B, Apache-2.0, and IBM's own numbers put the 8B at or above the 4.0 32B MoE on tool calling), Qwen3.5-9B, and Gemma 4 E2B/E4B. Every repo id and licence was verified against the Hugging Face API before being written down; Ministral 2512 is gated and was left out. yarn tool-routing:live runs the set against real models and prints an overall table plus a per-language one, so the numbers in a paper can be regenerated. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com> * Sample subsets by stride, and print every denominator The live runner's --limit took a prefix. The fixture file is ordered by scenario, so the first N tasks of a language are its first few scenarios rather than a sample of it, and an early run computed an absence rate over five items - the exact small-denominator problem the enlarged set exists to escape. Subsets now take an even stride, and every rate is printed with the number of examples it was taken over. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com> * Record the sub-10B run these changes were made to support Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Every module already produced runs and each stored them its own way, which is fine until someone asks what they ran yesterday, which cells failed, whether a configuration has already been run, and what commit produced a number. Those are questions about runs in general, so the registry holds one small record every kind shares and treats what the module did as opaque JSON. A new kind of run plugs in without the registry changing. Storage is configuration: an fs driver by default, one directory per run in the layout eval/tournaments already uses, and a sqlite driver for when there are thousands. node:sqlite ships with the runtime but is experimental, so it is imported only when configured. Filtering lives above both drivers and the test suite runs against each, because two stores that disagree about what a filter means would answer the same question differently depending on a setting. Provenance is captured rather than asked for - commit, dirty flag, params hash, runtime - and records machine facts only: the vote log already refuses to record who voted, and reintroducing identity beside it would undo that. Tool-routing sweeps from Compare are the first producer, through a helper whose failures are swallowed. Recording a run must never be able to break the run. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
A researcher's unit of work is a grid - these models over these datasets in these languages - long enough that something interrupts it, and re-running the finished cells wastes time and risks inconsistent numbers. The queue expands the grid into cells with stable keys, runs them under a global and per-group concurrency budget so one provider's rate limit is never tripped, retries the transient failure without retrying the broken cell forever, isolates a bad cell from the rest, and checkpoints each as it lands. Resume is a two-method Checkpoint interface: a Map in a test, the registry's append-only event log in production. Because it reads through the RunStore, resume works across processes - kill the job, restart it against the same run id, and only the unfinished cells run. That is what the event log was for. The scheduler is a fixed worker pool over a shared cursor; the worker holding the last slot of a saturated group drains it, so a single pass suffices and no cell is orphaned. The queue never reads params or summaries - a new kind of matrix needs no change to it, the same way a new run needs none to the registry. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
…on top (#12) The registry stores runs and the queue runs cells; a researcher wants one verb: run this grid, and if it dies, run it again and pick up where it stopped. runRegistryMatrix is that verb - it opens or reopens a run, checkpoints every cell to the run's event log, drives the queue, and finalises the status. A resume reopens the same run and reruns only the unfinished cells; a resume whose parameters hash differently is refused so it cannot become a different experiment under one id. The cell summaries now ride along in the checkpoint, so a results table is rebuilt from the registry even after a resume rather than living only in the process that produced it. First producer: yarn tool-routing:matrix, one cell per (model, language), resumable with --resume. Verified live through OpenRouter - interrupted a grid at 2/3 cells and resumed to run only the remainder under the same run id. Also fixed two usability bugs in yarn runs that this surfaced: it truncated run ids so you could not copy them, and it crashed with EPIPE when piped to head. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
An accuracy column cannot distinguish a model that routes correctly and formats badly from one that picks the wrong tool from one that acts when it should have declined - and those three need a parser fix, better tool descriptions, and something else again. This classifies them. Runs could not be triaged at all before, because only slice metrics survived: a run could say 6% failed to parse but never which items or why. So the registry gains artifacts - a run's full output, beside the summary rather than in it - and the tool-routing matrix stores each cell's report. Tool-routing examples now also record the expected outcome, because 'wrong arguments' is not a finding until the wanted ones are visible. The taxonomy is shared across harnesses on purpose, and ordered: the right tool inside the wrong key is an envelope failure, not a bare format one, since calling it format hides the most actionable result in the set. format, envelope and desynced are marked recoverable - a prompt or parser change rather than a better model. yarn failures triages a run from its artifacts without re-calling anything. On a live two-model run it found 10 of 13 failures were wrapper problems, and showed Ministral-3B inventing a cat joke and calling translate_text on it where BLOCK was owed. The request is printed without the tool catalogue, which is identical on every task and buries the line that differs. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Two accuracy numbers side by side invite one question that a point estimate cannot answer. Comparisons are now paired the way the evals are - every model answers the same items - with Wilson intervals rather than the normal approximation, which breaks at exactly the denominators these slices have, McNemar's exact test on the disagreements, a seeded paired bootstrap so an interval does not move on re-run, and Holm adjustment across pairs. Power warnings print next to the result, because the failure mode is not a wrong number but a true one that cannot carry the weight put on it. That earned itself immediately: a matrix reported llama-3.2-3b at 100% tool select against 90% for two others, which is an artefact of a denominator of 3 - the other 21 items never reached the metric because they failed to parse. On parse compliance the same run separates at 25% against 100%, p<0.001. Triage now produces something too. A cohort saves the current selection as a registry run - so it is listable, has provenance, and is a cell list the queue can consume - and re-running one calls only those items. Annotations let a reader overrule a derived classification while keeping what the classifier said, because one corrected the same way repeatedly is telling you to fix it. Two bugs the end-to-end run exposed, both mine: cohort cell keys kept a character artifact names forbid, so every re-run cell failed loudly; and the run listing's unit table was off by one, printing a two-minute-old run as '2s ago'. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Everything the CLI tools surface - the registry, failure triage, the paired comparison - was reachable only from a terminal. This puts it in the browser: browse runs, open one, see why it failed and whether a difference between models is real. A new top-level module, thin API routes over the harness logic that already existed, and a client app that is read-mostly - the only writes are a saved cohort and a hand correction, both of which the CLI can also make. The failures view is the side-by-side that ends an argument: the request, the expected outcome and the raw reply next to each other, kind chips that filter and separate the recoverable wrapper failures from the ones needing a better model, save-as-cohort, and reclassify. The compare view is the paired stats with Wilson intervals and Holm-adjusted p, and it reproduces the CLI finding exactly - llama at 100% tool-select over 2 items, 20% parsed with p=0.000. The run list polls, so a matrix started from the CLI visibly finishes on screen. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
The detail panel rendered after a long failure table, so opening one put the evidence off-screen - you had to scroll to find what you just clicked. It now renders above the table and scrolls itself into view on open, so a click shows the request, expectation and reply straight away. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Image comparison leaves the product surface for now behind one dormant switch; its suites, judge, APIs and adapters stay implemented for later wiring. With text as the only modality, Compare no longer asks the reader to choose one. The module nav now teaches the workflow: Generate, Compare, Analyze, Evolve, Deploy. Generate now hands Compare deterministic references, the bound verifier, metric, template revision, locale and seeds. Exact/numeric verifiers carry one reference; format-only verifiers remain unscored rather than inventing a gold. In a source checkout Generate also falls back to importing packages/generate directly via python3, so the browser's first step no longer requires pip install -e. Compare scores with the bound verifier - making 13.8000 a valid answer to numeric reference 13.8 at tolerance 1e-4 - while retaining flat gold for Expected/Asked/Got. Every completed text run becomes a compare-text registry run with a full redrob-text-eval/v1 artifact. The primary completion CTA deep-links to Analyze, which classifies wrong/partial/empty answers and compares models on paired pass rates with the same statistical safeguards as tool routing. Desktop gutters and card padding were increased; the old 16px edge read as no padding around dense tables. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Four random base36 characters collide about 1% of the time at 200 ids in a burst, which failed CI at random and, when it passed, still let two runs share a directory. A process-local counter from a random start makes a burst unique outright. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Merging main into develop after the tool-routing release kept both sides of the rewritten sections, so the file has carried two copies of the branching model, the squash rule and the release QA layers since. The remaining copy is byte for byte the one on main. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
…ING (#16) Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
Bump the five version fields, and start a changelog: 0.1.0 was never tagged, so v0.2.0 is the first release a reader can obtain by name. Co-authored-by: Janghoon Lee <savagemanage@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release merge,
release/0.2.0intomain. Merge this with a merge commit, never squash or rebase (CONTRIBUTING is explicit: a squash intomaindestroys the shared ancestry that keeps later merges conflict-free). After merging,maingets the annotated tagv0.2.0and this branch merges back intodevelop.mainis currently an ancestor of this branch, so the merge is clean with no conflicts.What is in it
0.1.0 could compare models and settle quality by human preference. 0.2.0 is about what happens after a number appears: where the run is recorded, how it resumes, why a model was wrong, and whether a difference between two models is real. Full detail is in the new CHANGELOG.md.
yarn runsyarn tool-routing:matrix, resume by run idyarn failuresThe release commit itself
0.2.0: root,apps/web,packages/harness,packages/tokenizers, andversion:inCITATION.cffCITATION.cffgainsdate-releasedCHANGELOG.mdadded, and linked from both READMEs. 0.1.0 was never tagged, sov0.2.0is the first release a reader can obtain by name rather than by pinning a commitQA
Layers 1 and 2 (automated), run on this branch:
yarn typecheck,yarn lint,yarn buildyarn test: 985/985, with no flake remaining (see Two defects found cutting the release: a colliding run id, a doubled CONTRIBUTING #16)verify:theme,verify:phase1,verify:gepa,verify:phase3,verify:custom-goal,verify:tournament,verify:tool-routing,verify:multi-turn,verify:preference-gen,generate:spec-types:check: all passLayer 3, by hand, the things CI structurally cannot do:
next start, notnext dev):/analyzeruns list, run detail, Failures tab with Expected/Asked/Got, Compare tab with rates and pair statistics, back navigation, plus/compare,/generate,/deploy,/evolve,/settings,/tool-routing. No JavaScript errors, no raw translation keys, no clipped controls. The only console output is Next.js CSS-preload timing warnings.verify:*scripts are deliberately offline, so that run is the live evidence.Known limitations, stated rather than hidden
COMPARE_IMAGE_UI_ENABLED = false. Nothing was deleted; suites, judges, adapters and tournament support remain in place.