Skip to content

feat(multimodal): support Qwen3 ASR and Omni #1905

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
slin1237 merged 5 commits into from
Jul 13, 2026
Merged

feat(multimodal): support Qwen3 ASR and Omni #1905

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5602d7a
feat(multimodal): support Qwen3 ASR and Omni
chenht2022 Jul 12, 2026
70d6b34
fix(ci): satisfy formatting and clippy checks
chenht2022 Jul 12, 2026
6a5af02
fix(grpc): box validation error responses
chenht2022 Jul 12, 2026
08e4d4f
ci: install FFmpeg for vLLM/torchcodec in e2e setup
lightseek-bot Jul 12, 2026
d56bd75
fix(multimodal): harden audio and placeholder handling
chenht2022 Jul 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions crates/grpc_client/proto/tokenspeed_encoder.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ package tokenspeed.grpc.encoder;
import "tokenspeed_scheduler.proto";

// TokenSpeed EPD encode-stage gRPC service. The gateway ships preprocessed
// multimodal tensors to a vision-tower-only encode worker; the worker runs the
// tower and pushes the resulting image embeddings to a prefill worker over
// multimodal tensors to an encoder-only worker; the worker runs the matching
// tower and pushes the resulting embeddings to a prefill worker over
// Mooncake, keyed per item by bootstrap_room. The gateway never sees the
// embeddings — it only triggers the encode and assigns each item its room.
//
Expand All @@ -24,7 +24,7 @@ message EncodeRequest {
string request_id = 1;

// This worker's assigned multimodal item — the same tensor shape the scheduler
// Generate path carries. The encode worker runs the vision tower on it; it
// Generate path carries. The encode worker runs the matching tower on it; it
// does not fetch or preprocess (that already happened in the gateway).
tokenspeed.grpc.scheduler.MultimodalInputs mm_inputs = 2;

Expand Down
12 changes: 6 additions & 6 deletions crates/grpc_client/proto/tokenspeed_scheduler.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ import "common.proto";
// TokenSpeed scheduler gRPC service. Self-contained wire definition apart
// from the cross-engine admin messages in smg.grpc.common (flush/profile).
// Trimmed to text+multimodal generation (no embed, no LoRA, no hidden-state
// forwarding). Multimodal carries preprocessed tensors only — image fetch +
// forwarding). Multimodal carries preprocessed tensors only — media fetch +
// per-model preprocess happen in the gateway (see crates/multimodal). EPD: a
// prefill request may carry `EncodeBootstrapInfo` so its image embeddings
// prefill request may carry `EncodeBootstrapInfo` so its multimodal embeddings
// arrive from an encode worker over Mooncake (see tokenspeed_encoder.proto).
service TokenSpeedScheduler {
rpc Generate(GenerateRequest) returns (stream GenerateResponse);
Expand Down Expand Up @@ -108,9 +108,9 @@ message GenerateRequest {
// Preprocessed multimodal payload. Absent for text-only requests.
MultimodalInputs mm_inputs = 9;

// EPD: present when this request's image embeddings arrive from an encode
// EPD: present when this request's multimodal embeddings arrive from an encode
// worker over Mooncake instead of being computed here. The worker waits for
// the embedding keyed by `bootstrap_room` rather than running the vision
// the embedding keyed by `bootstrap_room` rather than running the matching
// tower. Absent for non-disaggregated (aggregated) requests.
optional EncodeBootstrapInfo encode_bootstrap_info = 10;

Expand Down Expand Up @@ -142,11 +142,11 @@ message EncodeItemBootstrapInfo {
// The encode worker's bootstrap server (data-source side), not the prefill's.
string bootstrap_host = 2;
int32 bootstrap_port = 3;
// 63-bit rendezvous id: minted random per image by the gateway with no
// 63-bit rendezvous id: minted random per item by the gateway with no
// in-flight dedup, so the space must be wide enough that the birthday
// collision rate is negligible even with many independent gateways and
// thousands of concurrent rooms (int32's 2^31 collides every few days under
// load, silently cross-wiring one image's embedding onto another's room).
// load, silently cross-wiring one item's embedding onto another's room).
int64 bootstrap_room = 4;
}

Expand Down
2 changes: 2 additions & 0 deletions crates/multimodal/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ libloading = "0.8"
ndarray = "0.17"
once_cell = "1.21.4"
rayon = "1.12"
rustfft = "6.4"
symphonia = { version = "0.6", default-features = false, features = ["all"] }
Comment on lines +32 to +33

@coderabbitai coderabbitai Bot Jul 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Determine which Symphonia codecs/formats are actually exercised before FFmpeg fallback.
rg -n 'symphonia' crates/multimodal/src/audio/decode.rs -A5
rg -n 'Codec|CodecType|register_all|get_codecs|get_formats|probe' crates/multimodal/src/audio/decode.rs

Repository: lightseekorg/smg

Length of output: 4955

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n## Cargo.toml excerpt\n'
sed -n '1,80p' crates/multimodal/Cargo.toml

printf '\n## Symphonia decode implementation\n'
sed -n '84,170p' crates/multimodal/src/audio/decode.rs

printf '\n## Audio tests\n'
sed -n '470,540p' crates/multimodal/src/audio/decode.rs

Repository: lightseekorg/smg

Length of output: 7805

🏁 Script executed:

#!/bin/bash
python3 - <<'PY'
from pathlib import Path
p = Path("crates/multimodal/src/audio/decode.rs")
text = p.read_text()
# print only lines with Symphonia hint/codecs references and tests names for quick scan
for i, line in enumerate(text.splitlines(), 1):
    if "get_probe()" in line or "get_codecs()" in line or "Hint" in line or "TrackType" in line or "wav_i16_mono" in line or "symphonia_decodes_wav_to_mono_f32" in line:
        print(f"{i}: {line}")
PY

Repository: lightseekorg/smg

Length of output: 704

🏁 Script executed:

#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path("crates/multimodal/src/audio/decode.rs")
for start, end in [(84,170), (470,540)]:
    print(f"\n### lines {start}-{end}")
    for i, line in enumerate(p.read_text().splitlines(), 1):
        if start <= i <= end:
            print(f"{i:4}: {line}")
PY

Repository: lightseekorg/smg

Length of output: 6750

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n## Audio dispatch and suffix helpers\n'
sed -n '1,220p' crates/multimodal/src/audio/decode.rs

printf '\n## Search for ffmpeg/symphonia backend selection\n'
rg -n 'audio_temp_suffix|SMG_AUDIO_DECODE_BACKEND|ffmpeg fallback|decode_audio_with_ffmpeg|decode_audio_with_symphonia' crates/multimodal/src/audio/decode.rs

Repository: lightseekorg/smg

Length of output: 9407

Narrow Symphonia's feature set features = ["all"] pulls in every Symphonia codec/container reader, but this path only has WAV coverage and FFmpeg already handles the fallback path. If Symphonia only needs a smaller in-process subset, list those features explicitly instead of enabling everything.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/multimodal/Cargo.toml` around lines 32 - 33, Update the symphonia
dependency declaration in Cargo.toml to remove features = ["all"] and enable
only the explicit feature(s) required for the existing WAV handling path.
Preserve FFmpeg as the fallback and do not add unrelated codec or container
features.

opencv = { version = "0.99.0", default-features = false, features = ["clang-runtime", "imgproc", "videoio"], optional = true }
reqwest = { workspace = true, features = ["stream"] }
serde = { workspace = true, features = ["derive"] }
Expand Down
Loading
Loading