Judge requests forward image content blocks verbatim, so a text-only judge fails every attached request open to the capable tier

[himorishige]

Summary

The capability and custom judge modes send the judge the windowed
conversation messages exactly as the client sent them. A user message that
carries an image_url part therefore reaches the judge as a multimodal
content array. A judge is typically a small text-only model, and a text-only
OpenAI-compatible backend rejects that request with HTTP 400. The router
treats the 400 as an unavailable judge, folds it into None
(report_fail_open, reason upstream_non_5xx) and falls open to the capable
tier.

The effect is deterministic and silent: every request that carries an
attachment skips the judge and is served by the capable model, however easy
the task is. Nothing in the client-visible response says so; only the
switchyard_classifier_fail_open_total counter moves.

escalation mode does the opposite. summarize_for_judge → collect_text
drops Image / Audio / Video / File blocks with _ => {}, so that
judge sees a summary with no trace that an attachment existed. The judge
modes thus disagree on what a judge sees, and neither behavior is documented.

Measured on the released 0.2.0 binary with two judge backends (a hosted
text-only model on Fireworks, and a self-hosted fine-tuned model on vLLM
0.27.1). The code path is unchanged on main at bb011ca.

Where it happens

step	location (0.2.0 and main)	what happens to an Image block
inbound decode	switchyard-translation openai_chat/buffered.rs	image_url / input_image → ContentBlock::Image
judge window	libsy/src/algorithms/llm_class.rs trim_messages / task_messages	messages are cloned whole; blocks are not filtered
judge request	libsy/src/algorithms/util/llm_judge.rs StructuredJudge::build_request	windowed messages used as-is; only the system prompt is replaced
outbound encode	openai_chat/buffered.rs encode_openai_content	a user message with an image is emitted as a content array with the image_url part; only audio / video / unknown are turned into text
escalation summary	libsy/src/algorithms/util/escalation.rs collect_text	_ => {} — the block is dropped without a marker
judge failure	llm_judge.rs JudgeClassifier::verdict	any judge error → None → policy fallback (capable tier)

Reproduction (model-independent)

1. Run any OpenAI-compatible capture double that records request bodies and
   returns a fixed valid verdict, e.g.
   {"crux":"x","primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.95}.

2. Point the classifier, weak and strong targets of a capability
   route at that double (format = "openai_chat").

3. Send one chat completion through the route whose user message is a
   content array: one text part plus one image_url part (a small PNG
   data URI is enough).

4. The captured judge request contains the image:

   "messages": [
     {"role": "system", "content": "You are a task-level probability forecaster ..."},
     {"role": "user", "content": [
       {"type": "text", "text": "Summarize the issue in this screenshot"},
       {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KG..."}}
     ]}
   ],
   "response_format": {"type": "json_schema", ...}

   The request forwarded to the serving tier carries the same content
   array, which is correct for the model that answers.

5. Put a text-only model behind the judge target.

   Hosted (Fireworks, judge = deepseek-v4-flash-0731, strong_target =
   kimi-k3):

   WARN libsy: model call failed selected_model="accounts/fireworks/models/deepseek-v4-flash-0731"
        error=... upstream returned HTTP 400: {"error":{"type":"invalid_request_error",
        "message":"This model does not support image inputs"}}
   WARN libsy: judge verdict unavailable; routing without one
        judge_model="accounts/fireworks/models/deepseek-v4-flash-0731" reason="upstream_non_5xx"
   

   Self-hosted (vLLM 0.27.1, judge = a fine-tuned text-only Nemotron 3.5
   Lightning):

   HTTP 400 {"error":{"message":"judge is not a multimodal model","type":"BadRequestError","code":400}}
   

   In both cases the response comes from the capable tier and the routing
   log has no classifier row for the request. Controls on the same route:
   the same task text without the image is judged normally; a multimodal
   judge (qwen3p7-plus) on the same route returns a verdict. So the
   attachment itself is what the text-only judge rejects.

Why this matters

• The fail-open is the right safety choice for an outage. Here it is
  triggered by input shape, so the judge is effectively disabled for a whole
  class of traffic, and the operator only finds out from a metric.
• Clients that attach images routinely (chat gateways, coding agents that
  paste screenshots) lose the cost benefit of routing on exactly those
  requests.
• A judge fine-tuned on text-only conversations is the model least likely
  to accept image input, so the users most invested in the capability judge
  are the ones hit hardest.
• The judge modes should agree on what a judge sees. Today one forwards
  attachments and one deletes them without a trace.

Proposed direction

Give every judge a text projection of the windowed conversation, applied
once in the shared layer (StructuredJudge::build_request, or a helper that
every ClassifierInput goes through), so capability, custom and
escalation behave the same:

• Text / Refusal blocks are kept.
• Image / File / Audio / Video / Unknown blocks are replaced by a
  short placeholder such as [image attachment] or
  [file attachment: <media type>]. The judge still learns that an
  attachment is part of the task ("summarize this screenshot" and
  "summarize this" are different tasks), but no payload bytes are sent to
  it.
• ToolCall / ToolResult blocks keep today's handling, so the tool-pair
  window rule is unaffected.
• escalation's collect_text emits the same placeholder instead of
  dropping the block.

The request forwarded to the selected serving tier is untouched; only the
side call to the judge changes.

One open question for maintainers: should the projection be the default,
with an opt-out for people who run a multimodal judge and want it to see the
image (for example judge_input = "text" | "native" on the route, default
"text"), or should it be opt-in? My preference is default-on: sending a
payload the judge cannot consume should not be the default behavior, and an
operator who wants the judge to see images knows they have a multimodal
judge.

Out of scope for this issue: routing rules keyed on modality (a "pre-judge"
hard route for attachments) and any change to the fail-open policy. Both
are separate topics; this issue is only about giving the judge an input it
can always consume.

I can open a PR for the text projection if this direction is acceptable.

Environment

• switchyard-server 0.2.0 (crates.io), Docker, format = "openai_chat"
  targets; behavior confirmed unchanged on main at bb011ca
• Judge backends measured: Fireworks (deepseek-v4-flash-0731, text-only;
  qwen3p7-plus, multimodal) and vLLM 0.27.1 (fine-tuned text-only
  Nemotron 3.5 Lightning)
• Serving tiers in the affected deployment: both multimodal, so the
  attachment is legitimately useful to the model that answers, and the judge
  is the only hop that cannot take it

[ljefford2-cmyk]

One regression test worth adding: capture both the judge request and the serving request for the same input. Assert that text projection replaces attachments only in the judge request, while the serving request retains the original attachment content and ordering.

That would also catch accidental mutation if the two paths share request state. Running the same case through capability, custom, and escalation modes would help keep the projection contract consistent.