Feature/streaming redevelop from main

[NemesisGuy]

feat(streaming): add sentence-level streaming CLI with REPL, prompt cache, playback, and timings

🧩 Summary

This pull request introduces a sentence-level streaming inference CLI for SoulX Podcast with both one-shot and interactive REPL modes.
It also adds persistent prompt-feature caching, timing utilities, and a non-blocking compute → playback pipeline for smoother and faster generation.

Key Features

• Sentence-level streaming CLI
  • Supports one-shot and interactive --repl modes.
  • Supports --text, --speaker, --reference_wav, --play, --precache_prompts, --clear_cache, and --timings flags.
• Persistent prompt-feature cache
  • Caches extracted features (log-mel, speaker embeddings, flow mels) in .prompt_cache/ for reuse across runs.
  • Uses mtime validation to auto-refresh if source WAV changes.
• Non-blocking REPL
  • Producer-consumer design: compute and playback run in parallel threads.
  • Allows typing new input while the model processes the previous one.
  • Minimizes GPU idle gaps during playback.
• Timing and diagnostics
  • Adds timing.py with Timer and TIMING_COLLECTOR.
  • Use --timings to log per-stage durations (feature extraction, spk-embed, flow-mel, synthesis, playback).
• Improved CLI UX
  • Friendlier system messages, queued input confirmations, and clear status feedback.
• Updated documentation
  • Expanded README.md with streaming usage examples.
  • Added CHANGELOG.md and development helper for syntax validation.

---

🗂️ Files Changed (High-Level)

File	Description
cli/streaming.py	Streaming CLI with REPL, chunked playback, caching, and timing support.
soulxpodcast/utils/prompt_cache.py	New persistent feature cache (disk + memory) with mtime validation.
soulxpodcast/utils/timing.py	Timing helper with Timer context and collector.
soulxpodcast/utils/dataloader.py	Integrates cache + timing metrics.
readme.md	Updated with usage examples, playback notes, and minor typo fixes.
CHANGELOG.md	Added “Unreleased” entry describing the new streaming features.
tools/py_compile_all.py	Utility to compile and syntax-check all .py files.
requirements.txt	Playback libraries (sounddevice, simpleaudio) now included as default dependencies.

---

🎯 Motivation

This upgrade reduces perceived latency, improves GPU utilization, and enhances user experience for interactive and demo scenarios.

• Lower latency: Sentence-level chunking + cached features reduce end-to-end inference time.
• Non-blocking UX: Compute and playback overlap for smoother interactivity.
• Developer visibility: Timing instrumentation surfaces bottlenecks for further tuning.
• Reusability: Cached features persist across runs, avoiding repeated feature extraction.

---

🧪 Testing & Validation Checklist

1. Syntax Check

python .\tools\py_compile_all.py
# expect: "py_compile finished: OK"

2. Environment Setup

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

3. One-Shot Inference Test

python .\cli\streaming.py --model_path pretrained_models/SoulX-Podcast-1.7B --text "[S1]Hello world." --play

✅ Expect: audio playback or generated WAV output.

4. REPL Interactive Test

python .\cli\streaming.py --model_path pretrained_models/SoulX-Podcast-1.7B --repl --play

✅ Type multiple sentences. Observe queued confirmations and playback while typing the next input.

5. Timing Test

python .\cli\streaming.py --text "[S1]Quick test." --timings --model_path pretrained_models/SoulX-Podcast-1.7B --play

✅ Expect: timing breakdown per stage (feature_load, spk_embed, flow_mel, etc).

6. Cache Verification

# Run twice to confirm cached re-use
python .\cli\streaming.py --text "[S1]Hello again." --speaker S1 --model_path pretrained_models/SoulX-Podcast-1.7B --play

✅ Second run logs: [cache hit] using precomputed prompt features...
✅ Observe reduced runtime for repeated prompt features.

---

⚙️ Notes / Known Limitations

• Cache validation currently uses file mtime. Future improvement: add SHA1 hashing for content-based validation.
• Playback dependencies (sounddevice, simpleaudio) are installed by default.
• Runtime-only imports (e.g., torch, torchaudio, onnxruntime) may trigger IDE warnings but are correct for runtime usage.
• Process-based precompute not yet implemented — current design uses threaded producer/consumer pipeline.

---

🏷️ Labels & Milestone

labels: enhancement, cli, docs
milestone: next-release

---

✅ Benchmark / Performance Note

Prompt-feature caching reduces repeated feature extraction and shortens cold-to-warm turnaround time.
Exact timings depend heavily on hardware, GPU/CPU configuration, and whether the model is already loaded.

Reproducible Benchmark Instructions

1. Clear persisted prompt cache (optional, to simulate cold run):

rm -rf .prompt_cache/

2. Start REPL or run CLI (model loads once):

python ./cli/streaming.py --model_path pretrained_models/SoulX-Podcast-1.7B --repl --play --timings

3. Run a cold input (no cache) — observe TIMING_COLLECTOR output.

4. Run the same input again (warm) — observe cache hits and reduced feature_load duration.

5. Compare fields (ignore total: summation):

   • feature_load (cold vs warm)
   • model_forward (neural inference time)
   • total_run (end-to-end after model load)

⚠️ Example from one test environment: cold run ~5–6s total_run, warm run with cache hit ~3.8–5.7s.
Your results may vary based on GPU/CPU, model version, and system load.

---

Status: Ready for Review 🚀
Author: @NemesisGuy
Date: November 2025
Feature: Streaming CLI + Persistent Prompt Cache + REPL Playback