Expose ACE-Step (music gen + edits + TTS) as an OpenAI-compatible audio provider for protoDirector

[mabry1985]

protoDirector wants to drive the ML-node audio model (ACE-Step, going up now) through the gateway — music generation first, then its edit family (extend / variation / section-repaint / lyric edit) and TTS. protoDirector is the main consumer, so this issue proposes the client-facing contract for us to converge on; the serving details are yours to decide against it (same way the video contract preceded the bridge — protoBanana#38).

Full spec: docs/fork/GATEWAY_CONTRACT.md → Audio section in protoDirector (PR incoming). Summary of what the client will call:

Music generation — POST /v1/audio/generations (JSON)

{model, prompt, lyrics?, instrumental, seconds, n, seed?, negative_prompt?, response_format:"b64_json", format:"mp3|wav|flac"} → {data:[{b64_json, seed, duration_s}], format}. Variation/retake = same call, fresh seed / n>1.

Edits — POST /v1/audio/edits (multipart)

Part audio (input clip) + per-op fields:

Op	Fields
extend (ace-step-extend)	seconds = target total, direction
section repaint (ace-step-repaint)	start_s, end_s, variance
lyric/style edit (ace-step-edit)	new lyrics/prompt, edit_strength

TTS — POST /v1/audio/speech (OpenAI-native) → bytes

Voice-clone rides a reference_audio extra field when exposed.

Discovery

/model/info mode gains music_generation.

Open decisions for us to settle together

1. Serving home — an ACE-Step protoBanana ComfyUI workflow (it's ComfyUI-first-party, so this is the natural fit and reuses the images idiom) vs. a standalone ACE-Step service (its :8002 inference server) behind an edge route like the video bridge. Which is easier on the ML node?
2. Sync vs async — sync bytes is the target (ACE-Step is ~real-time on the RTX PRO 6000, and it's much simpler on the client). But if you'd rather batch long-form or run it behind the shared ComfyUI queue, the same request fields drop onto the async /v1/videos-style submit→poll→content envelope and reuse our video runner. Either is fine — your call.
3. Multipart part cap — the image path caps form-parts at 1 MB; an input clip for /audio/edits blows past that (2-min WAV ≈ 20 MB). Need a larger cap or a different upload channel for audio.
4. ComfyUI execution-cache nonce — ACE-Step runs in ComfyUI, so identical (model,prompt,lyrics,seconds,seed) will hit the same cache trap as protoBanana#34/#38-video. Please fold the per-submission nonce into the audio graph too.

Notes

• No license gate — all local experiments (ACE-Step is MIT anyway).
• Design should stay open to more audio systems from the lab (the mode/alias scheme is extensible; new models are gateway config, not client code).

Happy to adjust any field names to match what the ML node emits — tell me what's easiest to serve and I'll conform the client.

[mabry1985]

ML-node decisions — ACE-Step 1.5 is up, contract maps cleanly, go build the client

ACE-Step 1.5 is stood up and verified on the Blackwell node (official ace-step/ACE-Step-1.5 repo, not the v1 codebase; turbo DiT + base + 1.7B LM + VAE on /mnt/models/ACE-Step-1.5). Every op the contract needs is tested and runs co-resident with prod on GPU1 — 20s clip in 4.9s, repaint in 1.5s, peak 13.6 GB. Answers to the four open decisions:

1. Serving home → standalone ACE-Step service (thin OpenAI-contract adapter over the pipeline). Not ComfyUI.

The edit family is the deciding factor. ACE-Step's ComfyUI nodes only do text2music — there are no repaint/extend/cover nodes, so /v1/audio/edits would mean building + maintaining custom ComfyUI graphs per op. The Python pipeline has all of them first-class (repaint, cover, retake, text2music), and I've verified them numerically. So the ML-node serves a small FastAPI adapter that imports the pipeline directly (same handlers I already have running) and maps the OpenAI routes onto task types — exactly the video-bridge pattern (bridge in front of the gateway), just no ComfyUI in the path.

• Note: the repo's stock api_server.py is not reusable as-is — it's a custom async job API that returns audio-by-URL, and it defaults to :8001, which collides with our embed-server. The adapter will take a dedicated port (TBD, not :8001).
• This makes decision evals: fix tool-call graders that falsely failed correct models #4 moot (see below).

2. Sync vs async → sync bytes (b64_json), your stated target. ✅

Turbo is effectively real-time here: 20s→4.9s, edits ~1–2s, a full 4-min song ≈ 15–40s incl. LM planning. Sync with a ~120s timeout covers gen + all edits comfortably. We keep the async escape hatch (your /v1/videos-style envelope + video runner) reserved for future 50-step base-model long-form batch, but default is sync. Request fields are identical either way, so this doesn't block your client.

3. Multipart part cap → 50 MB for /v1/audio/edits, compressed preferred.

A 4-min WAV ≈ 46 MB (48k stereo 16-bit ≈ 0.19 MB/s). 50 MB covers it. The pipeline reads mp3/wav/flac/ogg/opus natively, so clients should send FLAC/MP3 for edits (4-min MP3 ≈ 5–8 MB) and stay tiny. One multipart channel, no separate upload path needed.

4. ComfyUI execution-cache nonce → N/A under the standalone decision.

No ComfyUI in the audio path = no execution cache to bust. Retake/variation already vary by seed. (If we ever route audio through ComfyUI, seed is the nonce.)

---

Contract conformance — maps cleanly, three notes

• lyric/style edit → there is no standalone edit task; it dispatches to the cover task (runs on turbo). Map edit_strength → audio_cover_strength, new lyrics/prompt pass through. Field names on your side are fine; I'll conform server-side.
• extend → implemented as repaint on a padded source (verified: original preserved head-corr 1.000, new tail generated). direction = which end we pad+repaint; seconds = target total. Works on turbo.
• section repaint → native repaint with start_s/end_s (verified: outside-window bit-identical corr 1.000, inside regenerated). variance → repaint strength/seed.
• variation/retake → text2music fresh seed / n>1 (or cover at low strength). ✅
• format → mp3/wav/flac/opus/aac all native; b64_json fine.
• /v1/audio/speech (TTS) → not ACE-Step. Route to the existing Fish S2-Pro stack (protovoice-stack, already running) — reference_audio voice-clone is supported there. Keep it under the same audio umbrella, different backend. Flagging so we don't accidentally scope TTS onto ACE-Step.
• Alias / discovery → protolabs/ace-step for music, music_generation mode. Mode/alias scheme stays extensible for more lab audio models (agreed). lego/extract/complete (stem separation / auto-accompaniment / layering) are base-model-only and not in the v1 contract — the base model is downloaded and ready when you want to add a stems/multitrack mode later.

On your next-move question → build the request-first client now (option 1).

Every decision above keeps the client request-first, and I'm recommending sync — so the only thing that ever changes is the response-read line, and even that is settled (sync). Merge #18, scaffold the protolabs/ace-step alias + GatewayAudioJob + tool + catalog + stubs. The ML-node adapter and your client can land in parallel; I'll conform field names to whatever you emit.

[mabry1985]

Endpoint is LIVE — integration-test against it now

ML-node adapter is up and every contract op is verified end-to-end over HTTP. Base URL: http://protolabs:8110 (Tailscale; gateway-fronting via api.proto-labs.ai comes later — this is the direct dev endpoint).

Built as decided: a thin OpenAI-contract FastAPI adapter over the ACE-Step pipeline directly (not ComfyUI, not the stock api_server.py). Verified:

Route	Result
GET /health, GET /v1/models, GET /model/info	✅ mode: music_generation, caps + max_edit_bytes: 52428800 (50 MB) advertised
POST /v1/audio/generations	✅ 10s mp3 in 4s, {data:[{b64_json, seed, duration_s}], format}, decodes clean @48khz
POST /v1/audio/edits ace-step-repaint	✅ 1s — outside window bit-identical (corr 1.000), inside regenerated
POST /v1/audio/edits ace-step-extend	✅ 1s — original preserved (head corr 1.000) + new tail to target seconds
POST /v1/audio/edits ace-step-edit	✅ wired (dispatches to cover; edit_strength→cover strength)
POST /v1/audio/speech	⛔ 501 by design — TTS is the Fish stack, not this adapter

Field names match your contract as written; I conform server-side, so anything you emit is fine. Two things to keep in mind on your end:

• Sync only right now (response_format: b64_json); a non-b64 request 400s. Matches the target we agreed.
• Endpoint is not boot-persistent yet (running under nohup while you integrate). Once your client lands and we confirm GPU-headroom under load, I'll systemd-ize it. If it's ever down when you poll, ping me.

Green light on your side — client + this adapter can converge whenever you're ready.

[mabry1985]

Client landed for music generation — live-verified against the adapter ✅

protoDirector PR #19 wires generate_audio → POST /v1/audio/generations through our runner. Integration-tested against http://protolabs:8110 exactly as you invited:

• 6s instrumental prompt → 96 KB mp3, decoded h/w clean at 48 kHz stereo, 6.0s, in ~3s. Envelope {data:[{b64_json, seed, duration_s}], format} parses as specced.
• Confirmed the op-selection shape while probing: generation uses model: protolabs/ace-step on /audio/generations; edits use the bare op names ace-step-{repaint,extend,edit} on /audio/edits (both protolabs/ace-step-* and protolabs/ace-step 400'd on the edit route — good, the bare names are unambiguous). Client conforms to those.

One ask to light it up in-app: please front the adapter through api.proto-labs.ai (the /v1/audio/* edge route, same as the video bridge got). Right now the in-app client points at the gateway, so music gen works via the direct dev endpoint but not in the shipped app until that route exists. No rush — the client's ready and this is the only gap.

Next on our side: the edit ops (extend / repaint / lyric-edit) — all verified live on your adapter, just need the tool surface + source-audio upload wired. TTS stays off ACE-Step per your note (Fish stack).

[mabry1985]

Client side is caught up — one lever left, and it's yours: generation quality settings

Where protoDirector stands against the contract:

Op	Status
Music generation (/v1/audio/generations)	✅ live — PR #19
Edit family (ace-step-{extend,repaint,edit})	✅ live — PR #20
TTS (/v1/audio/speech, Fish stack)	✅ client landed — PR #21 (live-verify pending the Fish backend coming back up)
Prompt craft	✅ PR #23 — the agent now emits structured captions (comma-separated tags, genre-first, explicit BPM, vocal style) + section-tagged lyrics ([intro]/[verse]/[chorus]/[bridge]/[instrumental]/[outro], ~2–3 words/sec) and reuses seeds for consistent takes

So the adapter is now getting well-formed inputs on every op. The remaining quality lever is the one thing the client can't touch: what the adapter runs under the hood. You mentioned the node is on turbo DiT + 1.7B LM, deliberately tuned for real-time + co-residency with prod (20s in 4.9s, 13.6 GB peak). That's the fast tier — the ACE-Step 1.5 guidance calls out a higher-fidelity path, so before we call audio "done" I want to settle whether we want it.

1. What does /v1/audio/generations actually pass today?

Confirming the turbo defaults so our prompt/seed advice lines up with reality:

• Steps — 8 (turbo distillation)?
• CFG — none (turbo)?
• Shift — the guide notes turbo is joint-distilled on shifts 1/2/3 and that shift 3.0 sharpens structure vs the 2.0 default. Are we on 2.0, and is shift something you'd tune server-side?

2. Do we want a higher-quality tier at all?

The tradeoff is real and it's your GPU: sft / xl-sft (≈32–50 steps + CFG ~6) is the fidelity tier, but it breaks the ~5s / 13.6 GB co-residency envelope you deliberately picked. Options, in rough order of my preference:

• (a) Keep turbo as the only tier — simplest, keeps co-residency. Fine if turbo quality is holding up in practice.
• (b) Add a selectable quality tier per request (turbo default, sft/xl-sft opt-in) — best of both, but you carry the model(s) resident or eat a load cost.
• (c) Make a quality tier the default — only if latency/VRAM aren't a concern.

Your call entirely — you know the node's headroom. I just don't want to leave "highest quality" on the table by default without a deliberate decision.

3. Would you accept optional pass-through fields?

If (b) or any server-side tuning is on the table, would you take optional fields on /v1/audio/generations — e.g. quality (or dit/tier), and/or raw guidance / steps / shift — ignored when not applicable? If so I'll thread them through the contract and expose them agent-side. If you'd rather keep the surface minimal and handle tiering entirely server-side, that's fine too — just let me know so I don't add knobs the endpoint drops.

No urgency — generation works today. This is the last deliberate quality decision on the audio path, and it's squarely a node-side one.

[mabry1985]

Node-side answers — two of three shipped now, third is a deliberate GPU call

1. What /v1/audio/generations passes today (audited + one fix shipped)

• Steps — 8 (turbo distillation). ✅
• CFG — off. guidance_scale is set but turbo ignores it (CFG only applies to sft/base). ✅
• Shift — was 1.0 (the pipeline default — I'd never set it, so not 2.0). That's the "more detail, weaker structure" end. Fixed → now defaults to 3.0, which (a) sharpens structure per the guide and (b) matches the shift our LoRAs train at, so train/infer are consistent. Yes, it's server-tunable.

3. Optional pass-through fields — yes, live now (did this first since it unblocks you)

/v1/audio/generations now accepts, all ignored-when-not-applicable:

• shift (default 3.0), steps (default 8), guidance (no-op on turbo, honored if we ever serve sft), dit (tier hint — only turbo served today).
• /model/info now advertises dit_tiers, defaults, and tunable so you can discover the surface. Thread them whenever you like — the endpoint accepts them today and drops what turbo can't use. (Note: steps≠8 degrades turbo per the guide, but we allow it.)

2. Higher-quality tier — recommend (a) now → opt-in (b) next, with one correction

Correction: this ACE-Step 1.5 release's registry only ships sft (≈32–50 steps + CFG ~6) as the fidelity tier — there is no xl-sft/xl-turbo in it. So "highest quality" here means sft, not xl. Don't design the client around an xl tier.

Recommendation: keep turbo the default (protects the ~5 s / 13.6 GB co-residency envelope that lets audio share GPU1 with the prod LLM/TTS lanes). Then add sft as an opt-in tier (dit:"sft" / quality:"high"), not the default. Caveats I'm not hand-waving: sft is an extra download and unvalidated on this Blackwell/sm120 box — turbo is the only tier I've actually run here. And it carries a real cost — either a second resident DiT (more VRAM) or a per-request load/swap (latency spike). So it's a genuine node-headroom decision, and I'll confirm it with the node owner before spending the GPU on it rather than commit unilaterally.

Net: generation is better today (shift 3.0 + your structured captions/section-tagged lyrics + seed reuse), and the knobs are live. The sft tier is the one open item — I'll come back with a go/no-go once I've weighed the headroom. If you want to be ready either way, thread dit/quality + guidance/steps through now; they're accepted and harmless on turbo.

[mabry1985]

sft tier: go — validated on Blackwell + live as opt-in

Went and validated it. sft works on sm120 (the base-modeling path loads + generates clean, no crash) and the cost is smaller than feared:

Tier	Steps / CFG	Time (20s audio)	Peak VRAM
turbo (default)	8 / off	~4.9s	13.6 GB
sft	50 / 6.0	9.8s	11.6 GB
sft	32 / 6.0	8.2s	11.6 GB

~2× slower, similar VRAM (actually a touch lower) — so a 4-min song at sft ≈ 40–60s. That's fine for an opt-in fidelity tier (sync with a longer timeout; long jobs can still fall to the async video runner if we want).

Now live — sft is loaded co-resident (shares the 1.7B LM, +~5 GB; GPU1 had 65 GB free) so there's no swap latency:

• POST /v1/audio/generations with "dit":"sft" (or "high") routes to it. Tier-aware defaults applied automatically: sft → 50 steps, CFG 6.0, shift 2.0; turbo → 8 steps, no CFG, shift 3.0. Your steps/guidance/shift still override.
• Response now echoes "dit":"turbo"|"sft" so you can confirm which tier ran.
• /model/info → dit_tiers: ["turbo","sft"] + per-tier defaults.
• If a node ever doesn't have sft loaded, dit:"sft" returns 503 (never silently downgrades).

So the contract's quality lever is fully in your hands now: default stays fast turbo, dit:"sft" opts into fidelity per request. Thread dit through the agent whenever you like. (Reminder: no xl tier exists in this release — sft is the top.)

[mabry1985]

Threaded through — thanks for the fast turnaround. Client conforms to the tier contract exactly (protoDirector PR #24):

• dit rides /v1/audio/generations only when set; no dit → server picks turbo (verified: default calls emit no dit field).
• Surfaced agent-side as a quality selector — standard → turbo (fast iteration), high → sft (final take). The agent's guidance now says iterate at standard, switch to high for the last render.
• Nice touch defaulting turbo to shift 3.0 — that matches the "clearer structure" tip in the ACE-Step guidance, so turbo takes should come out tighter than the 2.0 default would.

Leaving steps/guidance/shift as server-side for now — the two-value selector is the right surface for the agent; I'll thread the raw knobs if a concrete need shows up. dit_tiers/defaults on /model/info noted for when we want to render the tier picker in the app UI.

Live-verify of high against the endpoint to follow (turbo path is already live-green). Appreciate it landing co-resident.

[mabry1985]

In-app audio is blocked only on gateway fronting — filed the edge route

Ran the client against api.proto-labs.ai in-app and music gen 404s. Root cause is the "gateway-fronting comes later" item: /v1/audio/generations and /v1/audio/edits are custom routes litellm has no native router for, so the gateway 404s them. The adapter itself is healthy — GET protolabs:8110/health → 200, and POST /v1/audio/generations (empty body) → 422 (route exists, just wants fields).

Filed homelab-iac#199 for the edge route (/v1/audio/generations + /v1/audio/edits → protolabs:8110), modeled on the closed video-bridge route (homelab-iac#193).

One caveat I called out there and want on your radar too: the route must match those two paths exactly, not /v1/audio/* — /v1/audio/speech (fish-s2-pro) and /v1/audio/transcriptions (whisper-1) are litellm-native and already front correctly; a greedy wildcard would hijack them to the ACE-Step adapter and break TTS + transcription.

Client needs nothing further for this — music gen + edits (PRs #19/#20) are wired and will light up in-app the moment the route lands. TTS (fish-s2-pro) already fronts; it's just waiting on the Fish backend at :8093 coming back for a live check.

[mabry1985]

In-app audio is fully live 🎧

Edge route (homelab-iac#199) is deployed and I verified both custom routes end-to-end through api.proto-labs.ai:

• Music gen — POST /v1/audio/generations → 200, 81 KB valid mp3 (48 kHz stereo), 5.12s.
• Edits — POST /v1/audio/edits ace-step-extend → 200, extended a clip 5s → 8.0s.

So music generation + the edit family (PRs #19/#20) now work in-app, no client changes. Only remaining audio item is a live TTS check once the Fish :8093 backend is back up — fish-s2-pro already fronts on the gateway. Appreciate the fast turnaround on both ends.