Repository navigation
Add decision support (decide/choose/score) for Granite Switch models #1741
Copy link
Copy link
Open
Labels
area/adapter-functionsGranite adapter functions: framework and adaptiers including RAG, Guardian, CoreGranite adapter functions: framework and adaptiers including RAG, Guardian, CoreenhancementNew feature or requestNew feature or requestp2Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.
Description
Summary
Granite Switch models ship a decision adapter: a trained capability that answers fixed-answer questions (is this urgent? which team handles this? how severe, 0-2?) with a single token, read from the model's probability distribution over the answer options. One forward pass per question, fast enough to use inside control flow.
Mellea should expose this as plain Python.
Proposed API
from mellea.stdlib.components.intrinsic import decisions
# Yes/no - a probability you can threshold:
if decisions.decide(ticket, backend, "Does this require urgent attention?") >= 0.7:
escalate(ticket)
# Multiple choice - best option plus the full distribution:
route = decisions.choose(ticket, backend, "Which team handles this?",
{"payments": "Payout failures and refunds",
"account": "Login and account access"})
# Rating - expected level plus per-level probabilities:
sev = decisions.score(ticket, backend, "How urgent is this?",
["can wait a week", "this week", "today"])state (the first argument) is the thing being judged: a string, a dict or list, or a whole conversation. Answers are typed, and a distribution that doesn't concentrate on the declared options raises instead of returning a plausible-looking wrong number.
Context
- The decision adapter ships in the Granite Switch 4.1 checkpoints (announcement blog post)
- Serving requires vLLM with
--max-logprobs 20and the granite-switch package; also works with llama.cpp'sllama-server - The readout reads the model's top-20 token logprobs at the answer position and renormalizes over the option labels
Out of scope
- Accuracy/calibration benchmarking (model-team domain; needs a labeled evaluation set)
- A default revision pin table for Hub checkpoints
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area/adapter-functionsGranite adapter functions: framework and adaptiers including RAG, Guardian, CoreGranite adapter functions: framework and adaptiers including RAG, Guardian, CoreenhancementNew feature or requestNew feature or requestp2Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.
Activity