serving: release_id keys audits, quarantine and the READY pool; gateway routes by model - #1728
Merged
Conversation
…ay routes by model One model can have several blessed runtimes, and phase 1 assigns gpu -> release. Keying the audit window, quarantine, READY pool and served requests by model_id collides the moment two releases serve the same model, so everything the validator tracks moves to release_id (defaults to model_id, so a one-release loadout is unchanged). - loadout: release_id on ServingRelease, unique across the loadout; get() resolves a release_id then a model_id; an `audit` block per release overrides the logprob bands (a non-deterministic runtime ships its own; the constants stay the default). - gateway: /v1/models lists every release; `model` in the request body picks one (404 when it is not served); the request is audited against that release's reference. - miner: InferenceSynapse carries release_id and a miner serving another release refuses the call rather than answering for a release it was not blessed for. - window: strikes are counted per (hotkey, release) and persisted, so escalating quarantine is a policy change rather than a schema one. - store: audit_values/quarantine rename model_id -> release_id in place; existing rows carry over under their old key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One model can have several blessed runtimes, and phase 1 assigns
gpu -> release. Keying the audit window, quarantine, READY pool and served requests bymodel_idcollides the moment two releases serve the same model — so everything the validator tracks moves torelease_id. It defaults tomodel_id, so today's single-release loadout behaves identically.The bar this is aimed at: adding a second model (a 27B dense on the same sparkinfer runtime) should be a loadout entry plus a reference, not a refactor.
release_idonServingRelease, unique across the loadout;get()resolves arelease_idthen amodel_id. An optionalauditblock per release overrides the three logprob bands (a non-deterministic runtime ships its own; the constants stay the default)./v1/modelslists every release;modelin the request body picks one (404 when it is not served); the request is audited against that release's reference.InferenceSynapsecarriesrelease_id, and a miner serving another release refuses the call rather than answering for a release it was not blessed for.audit_values/quarantinerenamemodel_id->release_idin place and gainstrikes; existing rows carry over under their old key.Pairs with entrius/gittensor-db#NN (
release_idcolumns) and entrius/das-gittensor#NN — the DB columns should land first.CI: ruff, format, pyright, vulture, 742 tests green.