Harden audio transcription input handling#2597
Merged
Merged
Conversation
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
PR Summary by QodoHarden STT transcription input canonicalization and NumPy resampling
AI Description
Diagram
High-Level Assessment
Files changed (14)
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request hardens the audio transcription pipeline and adds design and implementation plans for the audio_cpp TTS provider. Key improvements include rejecting audio uploads when WAV conversion fails, canonicalizing compressed inputs to WAV for soundfile-backed providers (Canary, Qwen3 ASR, VibeVoice, and Parakeet MLX), and resampling non-16 kHz NumPy audio to 16 kHz for in-memory NeMo transcriptions. Feedback highlights a security issue in _ensure_wav_path_for_parakeet_mlx where existing .wav files bypass the base_dir containment check, potentially allowing path traversal.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Comment on lines
+404
to
+407
| def _ensure_wav_path_for_parakeet_mlx(audio_file_path: str, base_dir: Optional[Path] = None) -> str: | ||
| path_obj = Path(audio_file_path) | ||
| if path_obj.suffix.lower() == ".wav": | ||
| return str(path_obj) |
There was a problem hiding this comment.
If base_dir is provided, existing .wav files bypass the directory containment check because the function returns early on line 407. To prevent path traversal or unauthorized file access, validate the .wav path against base_dir using resolve_safe_local_path before returning it.
def _ensure_wav_path_for_parakeet_mlx(audio_file_path: str, base_dir: Optional[Path] = None) -> str:
path_obj = Path(audio_file_path)
if path_obj.suffix.lower() == ".wav":
if base_dir is not None:
safe_path = resolve_safe_local_path(path_obj, base_dir)
if safe_path is None:
raise STTTranscriptionError(
"Parakeet MLX audio loading failed: WAV path is outside the allowed directory."
)
return str(safe_path)
return str(path_obj)
Code Review by Qodo
Context used✅ Compliance rules (platform):
74 rules 1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Comment on lines
+171
to
+176
| ratio = float(target_sample_rate) / float(sample_rate) | ||
| new_len = max(1, int(round(len(audio_np) * ratio))) | ||
| x_old = np.linspace(0.0, 1.0, num=len(audio_np), endpoint=False, dtype=np.float32) | ||
| x_new = np.linspace(0.0, 1.0, num=new_len, endpoint=False, dtype=np.float32) | ||
| resampled = np.interp(x_new, x_old, audio_np) | ||
| return np.ascontiguousarray(resampled, dtype=np.float32), target_sample_rate |
There was a problem hiding this comment.
10. Unbounded resample length 🐞 Bug ➹ Performance
_prepare_numpy_audio_for_nemo’s linear interpolation fallback computes new_len directly from the sample-rate ratio with no upper bound or sample_rate sanity checks. For implausible/misreported sample rates, this can allocate extremely large arrays and cause high memory/CPU usage.
Agent Prompt
## Issue description
The linear resampling fallback in `_prepare_numpy_audio_for_nemo()` can produce extremely large arrays because `new_len` is unbounded and derived only from `(target_sr / sample_rate)`.
## Issue Context
The polyphase path mitigates many normal cases, but the fallback path will still run when SciPy is unavailable or ratios are large.
## Fix Focus Areas
- Add sample_rate validation (e.g., reject or clamp when `sample_rate` is outside a reasonable range like 8000–192000).
- Add a hard cap on `new_len` (or on multiplier) and fail fast with a clear error when exceeded.
- Consider capping by maximum supported audio duration in samples to prevent accidental/hostile allocations.
### References
- tldw_Server_API/app/core/Ingestion_Media_Processing/Audio/Audio_Transcription_Nemo.py[143-176]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tldw_Server_API/app/api/v1/endpoints/audio/audio_transcriptions.py (1)
720-782: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsolidate the four identical
invalid_audiorejections.The import-failure, conversion-failure, empty-output, and non-WAV-output branches all raise the same 400 payload with the same message. Extracting a small local helper removes ~30 lines of duplication and keeps the failure contract in one place.
♻️ Proposed helper
def _raise_invalid_audio(*, cause: Optional[Exception] = None) -> None: exc = HTTPException( status_code=status.HTTP_400_BAD_REQUEST, detail=_dictation_error_detail( http_status=status.HTTP_400_BAD_REQUEST, detail_status="invalid_audio", message="Audio file could not be converted to canonical WAV.", ), ) if cause is not None: raise exc from cause raise excEach branch then becomes a single
_raise_invalid_audio(cause=e)/_raise_invalid_audio()call after itslogger.debug(...).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tldw_Server_API/app/api/v1/endpoints/audio/audio_transcriptions.py` around lines 720 - 782, The audio transcription flow repeats the same invalid_audio 400 rejection in several branches around _convert_to_wav and the post-conversion validation, so extract a small local helper in audio_transcriptions.py to centralize the HTTPException construction and reuse it from each failure path. Keep the existing logger.debug calls, then replace each duplicated raise block with a single helper call such as _raise_invalid_audio(cause=e) for exception branches and _raise_invalid_audio() for empty/unusable output branches, preserving the same _dictation_error_detail payload and exception chaining behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@tldw_Server_API/app/core/Ingestion_Media_Processing/Audio/Audio_Transcription_Nemo.py`:
- Around line 143-178: Add direct unit tests for _prepare_numpy_audio_for_nemo
to cover its edge cases instead of relying only on
transcribe_with_canary/transcribe_with_parakeet. Create tests that call the
helper directly for stereo input downmixing (ndim > 1), the resample_poly path
guard when up/down exceed 1000, and the scipy ImportError/exception fallback
path in the resampling block. Use the function name
_prepare_numpy_audio_for_nemo as the main target so the tests stay stable if
call sites change.
- Around line 143-178: The early-return in _prepare_numpy_audio_for_nemo
incorrectly treats non-positive sample_rate as if the audio were already at
target_sample_rate. Split the condition so only sample_rate ==
target_sample_rate returns unchanged audio and handle sample_rate <= 0 as
invalid input in a separate branch, either by raising a clear error or by
avoiding any resampled-rate return. Make sure the return value from
_prepare_numpy_audio_for_nemo, and the downstream callers such as
_temp_wav_from_numpy, only label audio as target_sample_rate when it has
actually been resampled.
---
Outside diff comments:
In `@tldw_Server_API/app/api/v1/endpoints/audio/audio_transcriptions.py`:
- Around line 720-782: The audio transcription flow repeats the same
invalid_audio 400 rejection in several branches around _convert_to_wav and the
post-conversion validation, so extract a small local helper in
audio_transcriptions.py to centralize the HTTPException construction and reuse
it from each failure path. Keep the existing logger.debug calls, then replace
each duplicated raise block with a single helper call such as
_raise_invalid_audio(cause=e) for exception branches and _raise_invalid_audio()
for empty/unusable output branches, preserving the same _dictation_error_detail
payload and exception chaining behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: f75b45e3-86b6-4e24-b9d5-76c5c1444ef3
⛔ Files ignored due to path filters (3)
Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.mdis excluded by!docs/**Docs/superpowers/plans/2026-07-03-audio-transcription-pipeline-hardening-plan.mdis excluded by!docs/**Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.mdis excluded by!docs/**
📒 Files selected for processing (11)
backlog/tasks/task-12124 - Design-audio.cpp-TTS-provider-and-setup-integration.mdbacklog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.mdbacklog/tasks/task-12126 - Harden-audio-transcription-pipeline-input-handling.mdtldw_Server_API/app/api/v1/endpoints/audio/audio_transcriptions.pytldw_Server_API/app/core/Ingestion_Media_Processing/Audio/Audio_Transcription_Lib.pytldw_Server_API/app/core/Ingestion_Media_Processing/Audio/Audio_Transcription_Nemo.pytldw_Server_API/app/core/Ingestion_Media_Processing/Audio/stt_provider_adapter.pytldw_Server_API/tests/Audio/test_audio_transcriptions_adapter_path.pytldw_Server_API/tests/Audio/test_stt_provider_adapter.pytldw_Server_API/tests/Media_Ingestion_Modification/test_nemo_transcription.pytldw_Server_API/tests/Media_Ingestion_Modification/test_parakeet_mlx.py
Comment on lines
+143
to
+178
| def _prepare_numpy_audio_for_nemo( | ||
| audio_data: np.ndarray, | ||
| sample_rate: int, | ||
| *, | ||
| target_sample_rate: int = 16000, | ||
| ) -> tuple[np.ndarray, int]: | ||
| audio_np = np.asarray(audio_data, dtype=np.float32) | ||
| if audio_np.ndim == 0: | ||
| audio_np = audio_np.reshape(1) | ||
| elif audio_np.ndim > 1: | ||
| audio_np = np.mean(audio_np, axis=1).astype(np.float32, copy=False) | ||
| if audio_np.size == 0: | ||
| return np.ascontiguousarray(audio_np, dtype=np.float32), target_sample_rate | ||
| if sample_rate == target_sample_rate or sample_rate <= 0: | ||
| return np.ascontiguousarray(audio_np, dtype=np.float32), target_sample_rate | ||
|
|
||
| try: | ||
| from scipy import signal | ||
|
|
||
| divisor = math.gcd(int(sample_rate), int(target_sample_rate)) | ||
| up = int(target_sample_rate) // divisor | ||
| down = int(sample_rate) // divisor | ||
| if up <= 1000 and down <= 1000: | ||
| resampled = signal.resample_poly(audio_np, up, down) | ||
| return np.ascontiguousarray(resampled, dtype=np.float32), target_sample_rate | ||
| except _NEMO_RESAMPLE_EXCEPTIONS as exc: | ||
| logging.debug(f"NeMo numpy audio polyphase resample failed; using linear fallback: {type(exc).__name__}") | ||
|
|
||
| ratio = float(target_sample_rate) / float(sample_rate) | ||
| new_len = max(1, int(round(len(audio_np) * ratio))) | ||
| x_old = np.linspace(0.0, 1.0, num=len(audio_np), endpoint=False, dtype=np.float32) | ||
| x_new = np.linspace(0.0, 1.0, num=new_len, endpoint=False, dtype=np.float32) | ||
| resampled = np.interp(x_new, x_old, audio_np) | ||
| return np.ascontiguousarray(resampled, dtype=np.float32), target_sample_rate | ||
|
|
||
|
|
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Add direct unit tests for _prepare_numpy_audio_for_nemo.
This helper is only exercised indirectly via transcribe_with_canary/transcribe_with_parakeet at a single 8kHz→16kHz ratio. Edge cases such as stereo (ndim > 1) downmix, the up > 1000 or down > 1000 guard that skips resample_poly, and the scipy ImportError fallback path remain untested.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tldw_Server_API/app/core/Ingestion_Media_Processing/Audio/Audio_Transcription_Nemo.py`
around lines 143 - 178, Add direct unit tests for _prepare_numpy_audio_for_nemo
to cover its edge cases instead of relying only on
transcribe_with_canary/transcribe_with_parakeet. Create tests that call the
helper directly for stereo input downmixing (ndim > 1), the resample_poly path
guard when up/down exceed 1000, and the scipy ImportError/exception fallback
path in the resampling block. Use the function name
_prepare_numpy_audio_for_nemo as the main target so the tests stay stable if
call sites change.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Non-positive sample_rate is mislabeled as already resampled to target_sample_rate.
Line 156 combines two very different conditions: sample_rate == target_sample_rate (correctly no-op) and sample_rate <= 0 (invalid input). In the latter case, the raw un-resampled audio_np is returned but the function claims the audio is now at target_sample_rate. If this path is later hit by the fallback (_temp_wav_from_numpy(audio_np, model_sample_rate) at lines 553/642), the temp WAV would be tagged with an incorrect sample rate, silently corrupting playback speed/duration without raising any error.
🐛 Proposed fix
- if sample_rate == target_sample_rate or sample_rate <= 0:
- return np.ascontiguousarray(audio_np, dtype=np.float32), target_sample_rate
+ if sample_rate == target_sample_rate:
+ return np.ascontiguousarray(audio_np, dtype=np.float32), target_sample_rate
+ if sample_rate <= 0:
+ logging.debug(
+ f"NeMo numpy audio prep received non-positive sample_rate={sample_rate}; skipping resample"
+ )
+ return np.ascontiguousarray(audio_np, dtype=np.float32), sample_rate📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _prepare_numpy_audio_for_nemo( | |
| audio_data: np.ndarray, | |
| sample_rate: int, | |
| *, | |
| target_sample_rate: int = 16000, | |
| ) -> tuple[np.ndarray, int]: | |
| audio_np = np.asarray(audio_data, dtype=np.float32) | |
| if audio_np.ndim == 0: | |
| audio_np = audio_np.reshape(1) | |
| elif audio_np.ndim > 1: | |
| audio_np = np.mean(audio_np, axis=1).astype(np.float32, copy=False) | |
| if audio_np.size == 0: | |
| return np.ascontiguousarray(audio_np, dtype=np.float32), target_sample_rate | |
| if sample_rate == target_sample_rate or sample_rate <= 0: | |
| return np.ascontiguousarray(audio_np, dtype=np.float32), target_sample_rate | |
| try: | |
| from scipy import signal | |
| divisor = math.gcd(int(sample_rate), int(target_sample_rate)) | |
| up = int(target_sample_rate) // divisor | |
| down = int(sample_rate) // divisor | |
| if up <= 1000 and down <= 1000: | |
| resampled = signal.resample_poly(audio_np, up, down) | |
| return np.ascontiguousarray(resampled, dtype=np.float32), target_sample_rate | |
| except _NEMO_RESAMPLE_EXCEPTIONS as exc: | |
| logging.debug(f"NeMo numpy audio polyphase resample failed; using linear fallback: {type(exc).__name__}") | |
| ratio = float(target_sample_rate) / float(sample_rate) | |
| new_len = max(1, int(round(len(audio_np) * ratio))) | |
| x_old = np.linspace(0.0, 1.0, num=len(audio_np), endpoint=False, dtype=np.float32) | |
| x_new = np.linspace(0.0, 1.0, num=new_len, endpoint=False, dtype=np.float32) | |
| resampled = np.interp(x_new, x_old, audio_np) | |
| return np.ascontiguousarray(resampled, dtype=np.float32), target_sample_rate | |
| def _prepare_numpy_audio_for_nemo( | |
| audio_data: np.ndarray, | |
| sample_rate: int, | |
| *, | |
| target_sample_rate: int = 16000, | |
| ) -> tuple[np.ndarray, int]: | |
| audio_np = np.asarray(audio_data, dtype=np.float32) | |
| if audio_np.ndim == 0: | |
| audio_np = audio_np.reshape(1) | |
| elif audio_np.ndim > 1: | |
| audio_np = np.mean(audio_np, axis=1).astype(np.float32, copy=False) | |
| if audio_np.size == 0: | |
| return np.ascontiguousarray(audio_np, dtype=np.float32), target_sample_rate | |
| if sample_rate == target_sample_rate: | |
| return np.ascontiguousarray(audio_np, dtype=np.float32), target_sample_rate | |
| if sample_rate <= 0: | |
| logging.debug( | |
| f"NeMo numpy audio prep received non-positive sample_rate={sample_rate}; skipping resample" | |
| ) | |
| return np.ascontiguousarray(audio_np, dtype=np.float32), sample_rate | |
| try: | |
| from scipy import signal | |
| divisor = math.gcd(int(sample_rate), int(target_sample_rate)) | |
| up = int(target_sample_rate) // divisor | |
| down = int(sample_rate) // divisor | |
| if up <= 1000 and down <= 1000: | |
| resampled = signal.resample_poly(audio_np, up, down) | |
| return np.ascontiguousarray(resampled, dtype=np.float32), target_sample_rate | |
| except _NEMO_RESAMPLE_EXCEPTIONS as exc: | |
| logging.debug(f"NeMo numpy audio polyphase resample failed; using linear fallback: {type(exc).__name__}") | |
| ratio = float(target_sample_rate) / float(sample_rate) | |
| new_len = max(1, int(round(len(audio_np) * ratio))) | |
| x_old = np.linspace(0.0, 1.0, num=len(audio_np), endpoint=False, dtype=np.float32) | |
| x_new = np.linspace(0.0, 1.0, num=new_len, endpoint=False, dtype=np.float32) | |
| resampled = np.interp(x_new, x_old, audio_np) | |
| return np.ascontiguousarray(resampled, dtype=np.float32), target_sample_rate |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tldw_Server_API/app/core/Ingestion_Media_Processing/Audio/Audio_Transcription_Nemo.py`
around lines 143 - 178, The early-return in _prepare_numpy_audio_for_nemo
incorrectly treats non-positive sample_rate as if the audio were already at
target_sample_rate. Split the condition so only sample_rate ==
target_sample_rate returns unchanged audio and handle sample_rate <= 0 as
invalid input in a separate branch, either by raising a clear error or by
avoiding any resampled-rate return. Make sure the return value from
_prepare_numpy_audio_for_nemo, and the downstream callers such as
_temp_wav_from_numpy, only label audio as target_sample_rate when it has
actually been resampled.
rmusser01
force-pushed
the
codex/audio-transcription-pipeline-hardening
branch
from
fdc2e24 to
23bb566
Compare
rmusser01
force-pushed
the
codex/audio-transcription-pipeline-hardening
branch
from
461a324 to
8f5ecc6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Change summary
TODO: Human requester must replace this with the human-written change summary required by Docs/superpowers/AI_GENERATED_PR_CHANGE_SUMMARY_POLICY_2026_04_17.md before merge.
Test Plan
Summary by cubic
Hardens the audio transcription pipeline by enforcing canonical 16 kHz mono WAV at the endpoint and adapters, validating RIFF/WAVE headers, and normalizing NumPy inputs for NeMo. Removes fallbacks to compressed inputs and forces fresh conversions (
overwrite=True) to prevent spoofed or stale WAV reuse. Addresses TASK-12126.Bug Fixes
.wav, existence, and RIFF/WAVE header; usesoverwrite=True; returns 400invalid_audiowith no fallback.overwrite=Trueand enforce adapterbase_dir; Parakeet MLX uses_ensure_wav_path_for_parakeet_mlxto validate existing/converted paths stay insidebase_dirbefore buffered loading._prepare_numpy_audio_for_nemodownmixes to mono float32 and resamples to 16 kHz; validates sample rate range; guards large ratios; uses polyphase when available and bounded linear fallback otherwise; handles empty/scalar arrays.overwrite=True), MLXbase_dir, and NeMo resampling with SciPy/polyphase failure fallbacks; focused suite: 85 passed, 2 skipped.Migration
invalid_audiowhen uploads cannot be converted to WAV. Upload canonical WAV or a decodable format.Written for commit 8f5ecc6. Summary will update on new commits.