Skip to content

Add decision support (decide/choose/score) for Granite Switch models #1741

Open

Description

@planetf1

Summary

Granite Switch models ship a decision adapter: a trained capability that answers fixed-answer questions (is this urgent? which team handles this? how severe, 0-2?) with a single token, read from the model's probability distribution over the answer options. One forward pass per question, fast enough to use inside control flow.

Mellea should expose this as plain Python.

Proposed API

from mellea.stdlib.components.intrinsic import decisions

# Yes/no - a probability you can threshold:
if decisions.decide(ticket, backend, "Does this require urgent attention?") >= 0.7:
    escalate(ticket)

# Multiple choice - best option plus the full distribution:
route = decisions.choose(ticket, backend, "Which team handles this?",
                         {"payments": "Payout failures and refunds",
                          "account": "Login and account access"})

# Rating - expected level plus per-level probabilities:
sev = decisions.score(ticket, backend, "How urgent is this?",
                      ["can wait a week", "this week", "today"])

state (the first argument) is the thing being judged: a string, a dict or list, or a whole conversation. Answers are typed, and a distribution that doesn't concentrate on the declared options raises instead of returning a plausible-looking wrong number.

Context

  • The decision adapter ships in the Granite Switch 4.1 checkpoints (announcement blog post)
  • Serving requires vLLM with --max-logprobs 20 and the granite-switch package; also works with llama.cpp's llama-server
  • The readout reads the model's top-20 token logprobs at the answer position and renormalizes over the option labels

Out of scope

  • Accuracy/calibration benchmarking (model-team domain; needs a labeled evaluation set)
  • A default revision pin table for Hub checkpoints

Activity

  1. self-assigned this
    on Oct 16, 2026
  2. added
    enhancementNew feature or request
    p2Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.
    area/adapter-functionsGranite adapter functions: framework and adaptiers including RAG, Guardian, Core
    on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/adapter-functionsGranite adapter functions: framework and adaptiers including RAG, Guardian, CoreenhancementNew feature or requestp2Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions