Skip to content

Commit 0393fcb

Browse files
committed
Adapter benchmark: a Classifier column, for granite-switch's classifier slots
A classifier slot (granite-switch PR #134) holds a linear head instead of an adapter: its control token, placed last, makes the model emit one label word, the head's verdict on the final hidden state there. No row measured one. A fourth technology, classifier, does: one head per intrinsic that has a classifier entry in adapters.yaml (answerability and guardian-core, labels yes / no; answerability's read as "answerable" / "unanswerable", which are not single tokens). - Staging: a head folder (classifier_head.safetensors, classifier.json with its labels in row order) is detected and staged like an adapter, with its checksum and labels in the fingerprint; labels not the intrinsic's, in order, are refused. - Compose: heads join the LoRA / aLoRA checkpoint, with their labels, after checks that a head is the base's width and its labels single tokens. The PR renamed classifier slots to activation probes in aa7dd88 (manifest kind probe, head file probe_head.safetensors); the run asks the commit's composer which name it uses, gives a probe composer a linked copy of the head, and records the kind. A group of heads alone gets a filler LoRA whose delta is zero, since neither composer can compose heads without an adapter. A commit whose composer has neither skips the cells. - Generate: classifier cells render with adapter_name (both PR versions take it for a head), must end in the head's marker, and generate one token, in an eager engine of their own (FULL CUDA-graph replay of a classifier slot differs from eager under vLLM 0.26, granite-switch 1a78c29). Scoring maps the label to the scorer's answer. - Reference: the same head on the HF base model's final hidden state. - Page: 13 columns per intrinsic; "—" where an intrinsic has no head, on every row, in the PR comment's table too. The serving blocks measure no classifier: it decodes nothing. Releases hold none. - --only takes intrinsic/technology, and its results merge cell by cell, so the column can be added to an existing row. No bench_version or reference_version bump: the other cells are unchanged, and old rows show the new cells as not run. Signed-off-by: AlonMalach <alonmalach@gmail.com>
1 parent 2506b7c commit 0393fcb

13 files changed

Lines changed: 1244 additions & 117 deletions

File tree

‎benchmarks/adapter_eval/adapters.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,14 @@ technologies:
216216
- id: sr
217217
label: SR
218218
source: "Shadow Residual (q,o + MLP, r=32, cross r=32, shared KV)"
219+
# A classifier slot: a linear head on the frozen base model's final hidden
220+
# state at the end of the prompt, composed into the LoRA / aLoRA checkpoint,
221+
# its verdict emitted as one label word. Only intrinsics with a `classifier`
222+
# entry have one; it answers in one token, so it has no serving blocks.
223+
- id: classifier
224+
label: Classifier
225+
source: "Linear head on the final hidden state (frozen base, logistic regression)"
226+
serving: false
219227

220228
# Decode throughput, the Throughput columns: per technology, granite-switch
221229
# against stock vLLM serving the same adapters (native-lora), measured by the
@@ -278,6 +286,12 @@ intrinsics:
278286
headline: accuracy
279287
headline_label: Accuracy
280288
max_new_tokens: 200
289+
# Its classifier head's labels, in the head's row order, and each as the
290+
# scorer reads an answer. A label must be one token of the base model's
291+
# tokenizer, and "answerable" / "unanswerable" are several.
292+
classifier:
293+
labels: ["yes", "no"]
294+
to_answer: {"yes": '"answerable"', "no": '"unanswerable"'}
281295
- id: query_clarification
282296
name: Query clarification
283297
scorer: query_clarification
@@ -290,6 +304,10 @@ intrinsics:
290304
headline: accuracy
291305
headline_label: Accuracy
292306
max_new_tokens: 24
307+
# yes: the text is a risk; no: it is not. The scorer reads them as they are.
308+
classifier:
309+
labels: ["yes", "no"]
310+
to_answer: {"yes": "yes", "no": "no"}
293311
- id: requirement_check
294312
name: Requirement check
295313
scorer: requirements

‎benchmarks/adapter_eval/common.py‎

Lines changed: 63 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -39,13 +39,23 @@
3939
PYPI_VERSION = re.compile(r"\d+(\.\d+)*")
4040

4141
# Technologies composed into the same checkpoint. SR is a whole-checkpoint
42-
# dual-stream mode and the composer refuses to mix it with LoRA / aLoRA.
43-
COMPOSE_GROUPS = {"single_stream": ("lora", "alora"), "dual_stream": ("sr",)}
42+
# dual-stream mode and the composer refuses to mix it with LoRA / aLoRA. A
43+
# classifier slot (a linear head, no LoRA) rides in the LoRA / aLoRA
44+
# checkpoint: the composer cannot compose classifiers on their own, and its
45+
# marker selects the head without changing the active adapter.
46+
CLASSIFIER = "classifier"
47+
COMPOSE_GROUPS = {
48+
"single_stream": ("lora", "alora", CLASSIFIER),
49+
"dual_stream": ("sr",),
50+
}
4451

4552
# Reference columns (``reference.py``): each technology's staged checkpoint
46-
# with HF + PEFT and no granite-switch, and the base model with no adapter.
53+
# with HF + PEFT (a classifier: its head on the HF base model) and no
54+
# granite-switch, and the base model with no adapter.
4755
BASE_COLUMN = "base"
48-
REFERENCE_COLUMNS = ("lora", "alora", "sr", BASE_COLUMN)
56+
REFERENCE_COLUMNS = ("lora", "alora", "sr", CLASSIFIER, BASE_COLUMN)
57+
# The skip reason of an intrinsic with no classifier head (adapters.yaml).
58+
NO_CLASSIFIER = "no classifier head for this task"
4959
# Libraries whose versions the reference run records.
5060
REFERENCE_LIBRARIES = ("torch", "transformers", "peft")
5161

@@ -144,6 +154,25 @@ class Technology:
144154
id: str
145155
label: str
146156
source: str
157+
# Whether the serving blocks (throughput, switching) measure it. A
158+
# classifier answers in one token: there is nothing to decode or switch.
159+
serving: bool = True
160+
161+
162+
@dataclass(frozen=True)
163+
class Classifier:
164+
"""An intrinsic's classifier head (``classifier`` in adapters.yaml): its
165+
label words, in the head's row order, and what each means to the scorer."""
166+
167+
# Single-token words: the head's verdict is emitted as one of them.
168+
labels: tuple[str, ...]
169+
# Each label as the intrinsic's scorer reads an answer, e.g. "yes" ->
170+
# '"answerable"' for answerability, whose own words are several tokens.
171+
to_answer: dict
172+
173+
def answer(self, label: str) -> str:
174+
"""The scorer's form of a generated label word; any other text as is."""
175+
return self.to_answer.get(label.strip(), label)
147176

148177

149178
@dataclass(frozen=True)
@@ -157,6 +186,30 @@ class Intrinsic:
157186
# Bumped when only this intrinsic's scoring changes: its saved answers are
158187
# scored again (rescore.py) instead of generated again.
159188
score_version: int = 1
189+
# Its classifier head, when it has one; None skips its classifier cell.
190+
classifier: Classifier | None = None
191+
192+
193+
def make_intrinsic(raw: dict) -> Intrinsic:
194+
"""An ``intrinsics`` entry, its ``classifier`` checked: two or more labels,
195+
each mapped to the scorer's answer."""
196+
raw = dict(raw)
197+
head = raw.pop("classifier", None)
198+
if head is not None:
199+
labels = tuple(str(label) for label in head.get("labels") or ())
200+
to_answer = {str(k): str(v) for k, v in (head.get("to_answer") or {}).items()}
201+
if (
202+
set(head) - {"labels", "to_answer"}
203+
or len(labels) < 2
204+
or len(set(labels)) != len(labels)
205+
or set(to_answer) != set(labels)
206+
):
207+
raise ValueError(
208+
f"intrinsic {raw.get('id')}: a classifier needs two or more "
209+
"distinct labels, each with its answer in to_answer"
210+
)
211+
head = Classifier(labels=labels, to_answer=to_answer)
212+
return Intrinsic(**raw, classifier=head)
160213

161214

162215
@dataclass(frozen=True)
@@ -281,6 +334,11 @@ def public_models(self) -> tuple[Model, ...]:
281334
"""The models the results page shows."""
282335
return tuple(m for m in self.models if not m.private)
283336

337+
@property
338+
def serving_technologies(self) -> tuple[Technology, ...]:
339+
"""The technologies the serving blocks (throughput, switching) measure."""
340+
return tuple(t for t in self.technologies if t.serving)
341+
284342
def source(self, tech_id: str) -> str:
285343
"""How the model's adapters of a technology were trained, for the page."""
286344
own = self.model.technologies.get(tech_id, {}).get("source")
@@ -397,7 +455,7 @@ def load_spec(path: Path = SPEC_PATH, model: str | None = None) -> Spec:
397455
reference_version=int(raw["reference_version"]),
398456
models=models,
399457
technologies=tuple(Technology(**t) for t in raw["technologies"]),
400-
intrinsics=tuple(Intrinsic(**i) for i in raw["intrinsics"]),
458+
intrinsics=tuple(make_intrinsic(i) for i in raw["intrinsics"]),
401459
model_id=ids[0],
402460
throughput=Throughput(**raw.get("throughput", {})),
403461
switching=Switching(**raw.get("switching", {})),

‎benchmarks/adapter_eval/generate.py‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,12 @@
2929
``adapter_name``, in the job's document style and chat-template options
3030
(``prompts.py``). An unknown name silently renders the base-model prompt, so
3131
every prompt is checked to contain the adapter's control token.
32+
33+
A classifier job (``"classifier": true``) names its head the same way: the
34+
template drops the generation prompt and appends the head's marker as the
35+
last token, where the head reads. Its prompts must end in that marker; the
36+
verdict is the first generated token, the label word. (granite-switch PR #134
37+
also took ``classifier_name`` up to 861b917; ``adapter_name`` works in both.)
3238
"""
3339

3440
from __future__ import annotations
@@ -62,6 +68,10 @@ def render(tok, row: dict, job: dict, control_token_id: int | None) -> list[int]
6268
ids = list(tok(text, add_special_tokens=False).input_ids)
6369
if control_token_id is not None and control_token_id not in ids:
6470
raise ControlTokenMissing(f"control token id for {adapter_name} not in prompt")
71+
if job.get("classifier") and (
72+
control_token_id is None or ids[-1] != control_token_id
73+
):
74+
raise ControlTokenMissing(f"classifier marker for {adapter_name} is not last")
6575
return ids
6676

6777

@@ -189,7 +199,10 @@ def run(spec: dict) -> dict:
189199
completion = out.outputs[0]
190200
row["generated_content"] = completion.text
191201
row["prompt_tokens"] = len(prompt["prompt_token_ids"])
192-
truncated += completion.finish_reason == "length"
202+
# A classifier's one-token answer always stops at its length.
203+
truncated += completion.finish_reason == "length" and not job.get(
204+
"classifier"
205+
)
193206
write_jsonl(Path(job["out_path"]), kept)
194207
status["jobs"][key].update(
195208
n_generated=len(kept),

‎benchmarks/adapter_eval/hf_generate.py‎

Lines changed: 84 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,11 @@
2525
pod at a pinned commit) with the checkpoint loaded by PEFT, as that repo's
2626
own generation does. The checkpoint's invocation tokens were dropped first
2727
(``staged.peft_sr_copy``).
28+
* ``classifier``: the base model and the staged head, no generation. Each
29+
prompt is the conversation without a generation prompt, then the base's
30+
role-open marker (``prompts.classifier_ids``), where a composed model's
31+
classifier marker is embedded; the head reads the final hidden state there
32+
and its best-scoring label is the answer.
2833
2934
Prompts are rendered by the base tokenizer's chat template, with the same
3035
documents, tools, document style and chat-template options as the
@@ -59,8 +64,15 @@
5964
import traceback
6065
from pathlib import Path
6166

62-
from .prompts import chat_text
63-
from .staged import CONFIG_FILE, WEIGHTS_FILE, read_jsonl, write_jsonl
67+
from .prompts import chat_text, classifier_ids
68+
from .staged import (
69+
CONFIG_FILE,
70+
HEAD_FILE,
71+
WEIGHTS_FILE,
72+
classifier_meta,
73+
read_jsonl,
74+
write_jsonl,
75+
)
6476

6577

6678
class CellFailed(Exception):
@@ -250,11 +262,81 @@ def generate_rows(model, tok, prompts: list[list[int]], budgets: list[int], job:
250262
return outputs, truncated
251263

252264

265+
def classify_rows(model, head: dict, prompts: list[list[int]], job: dict) -> list[int]:
266+
"""Each prompt's best-scoring head row: the head (``weight [labels, hidden]``,
267+
``bias``) on the base model's final hidden state at the prompt's last
268+
token. One prompt per forward pass: no padding to change the numbers."""
269+
import torch
270+
271+
weight = head["weight"].to("cuda", torch.float32)
272+
bias = head["bias"].to("cuda", torch.float32)
273+
picks = []
274+
for k, ids in enumerate(prompts):
275+
with torch.no_grad():
276+
hidden = model.model(torch.tensor([ids], device="cuda")).last_hidden_state
277+
scores = weight @ hidden[0, -1].float() + bias
278+
picks.append(int(scores.argmax()))
279+
if (k + 1) % 500 == 0:
280+
print(f"[hf_generate] {job['key']}: {k + 1} / {len(prompts)}", flush=True)
281+
return picks
282+
283+
284+
def run_classifier(job: dict) -> dict:
285+
"""A classifier reference cell: the staged head on the HF base model."""
286+
import torch
287+
from safetensors.torch import load_file
288+
from transformers import AutoTokenizer
289+
290+
key = job["key"]
291+
head_dir = Path(job["adapter_dir"])
292+
labels = classifier_meta(head_dir)["labels"]
293+
head = load_file(str(head_dir / HEAD_FILE))
294+
tok = AutoTokenizer.from_pretrained(job["base_model"])
295+
rows = read_jsonl(Path(job["eval_path"]), job.get("limit"))
296+
kept, prompts, too_long = [], [], 0
297+
for row in rows:
298+
ids = classifier_ids(
299+
tok,
300+
row,
301+
job.get("documents", "native"),
302+
job.get("chat_template_kwargs", {}),
303+
)
304+
if len(ids) > int(job["max_model_len"]):
305+
too_long += 1
306+
continue
307+
kept.append(row)
308+
prompts.append(ids)
309+
if not kept:
310+
raise CellFailed("no rows left to classify")
311+
t0 = time.time()
312+
with load_lock(job.get("load_lock")):
313+
model = load_model({**job, "column": "base"})
314+
print(f"[hf_generate] {key}: model loaded in {time.time() - t0:.0f}s", flush=True)
315+
t0 = time.time()
316+
for row, ids, pick in zip(kept, prompts, classify_rows(model, head, prompts, job)):
317+
row["generated_content"] = labels[pick]
318+
row["prompt_tokens"] = len(ids)
319+
write_jsonl(Path(job["out_path"]), kept)
320+
seconds = round(time.time() - t0, 1)
321+
print(f"[hf_generate] {key}: {len(kept)} rows in {seconds:.0f}s", flush=True)
322+
return {
323+
"ok": True,
324+
"n_rows": len(rows),
325+
"too_long": too_long,
326+
"n_generated": len(kept),
327+
"truncated": 0,
328+
"seconds": seconds,
329+
"gpu": torch.cuda.get_device_name(0),
330+
}
331+
332+
253333
def run(job: dict) -> dict:
254334
import torch
255335
from safetensors.torch import load_file
256336
from transformers import AutoTokenizer
257337

338+
if job["column"] == "classifier":
339+
return run_classifier(job)
258340
key = job["key"]
259341
max_model_len = int(job["max_model_len"])
260342
tok = AutoTokenizer.from_pretrained(job["base_model"])

‎benchmarks/adapter_eval/prompts.py‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -148,9 +148,11 @@ def chat_text(
148148
documents: str,
149149
template_kwargs: dict,
150150
instruction: dict | None = None,
151+
generation_prompt: bool = True,
151152
**extra,
152153
) -> str:
153-
"""A row's prompt, ending in the generation prompt.
154+
"""A row's prompt, ending in the generation prompt (unless
155+
``generation_prompt`` is False: a classifier head's prompt).
154156
155157
``documents`` is the row's document style, ``template_kwargs`` its model's
156158
chat-template options, ``instruction`` a base-model instruction
@@ -173,8 +175,30 @@ def chat_text(
173175
messages,
174176
tools=row.get("tools"),
175177
documents=documents_arg,
176-
add_generation_prompt=True,
178+
add_generation_prompt=generation_prompt,
177179
tokenize=False,
178180
**template_kwargs,
179181
**extra,
180182
)
183+
184+
185+
# The token a granite-switch classifier marker is embedded as: the base
186+
# template's role-open marker, the first token of its generation prompt. The
187+
# composed model's template drops the generation prompt and appends the
188+
# marker last, so a head reads the hidden state at this token.
189+
ROLE_OPEN = ("<|start_of_role|>", "<|im_start|>")
190+
191+
192+
def classifier_ids(
193+
tokenizer, row: dict, documents: str, template_kwargs: dict
194+
) -> list[int]:
195+
"""A row's prompt as a classifier head reads it on the base model: the
196+
conversation without a generation prompt, then the role-open marker."""
197+
text = chat_text(
198+
tokenizer, row, documents, template_kwargs, generation_prompt=False
199+
)
200+
vocab = tokenizer.get_vocab()
201+
marker = next((m for m in ROLE_OPEN if m in vocab), None)
202+
if marker is None:
203+
raise ValueError("the tokenizer has no role-open marker")
204+
return [*tokenizer(text, add_special_tokens=False).input_ids, vocab[marker]]

0 commit comments

Comments
 (0)