Skip to content

test(sft): capture warnings before prefetch - #358

Merged
SigureMo merged 1 commit into
redai-studio:mainfrom
ShigureNyako:session/sess_dev_neko_fix_relax_streaming_dataset_warning_capture_race
Sep 23, 2026
Merged

SigureMo merged 1 commit into
redai-studio:mainfrom
ShigureNyako:session/sess_dev_neko_fix_relax_streaming_dataset_warning_capture_race

Conversation

@ShigureNyako

Copy link
Copy Markdown
Contributor

What

Fix the warning-capture race in test_streaming_dataset_missing_media_skip_refills_batch. Only tests/engine/sft/dataset/test_streaming.py changes: move one call and explain its ordering.

Why

Follow-up requested in #273. Verified against current main (c7d2847378cdf24c99b22aadb4b2fd078ff225a0): shuffle() calls PrefetchBuffer.set_index_order(), which starts background processing immediately. The missing-image warning can therefore be emitted before the test installs its capture callback.

How

Move ds.shuffle(0) after the logger.warning monkeypatch, inside the existing try/finally. This also keeps prefetch startup covered by ds.stop() cleanup. Preserve all existing inline/async/prefetch cases and assertions. No production changes, sleeps, retries, or new dependencies.

Testing

Local macOS arm64, Python 3.10.21:

  • pre-commit run --all-files --show-diff-on-failure — all hooks passed using Python 3.10.
  • python -m pytest tests/engine/sft/dataset/test_streaming.py -q — 57 passed.
  • python -m pytest tests/engine/sft/dataset/ tests/utils/data/test_streaming_dataset.py -q -rs — 142 passed, 1 skipped (optional Parquet test: pyarrow unavailable).
  • Full suite: python -m pytest tests/ -x -q --tb=short stopped during unrelated collection because fastapi is unavailable (23 skipped, 1 error). GPU/multi-node integration validation was not run: this host has no required GPU cluster.
  • Additional tests: not needed; the existing three-mode regression test retains its assertions.
  • Documentation: not applicable to this test-only fix.

Type of Change

  • Bug fix (test-only)

Screenshots / Logs

A temporary local-only pytest probe wrapped the real shuffle() and joined its real prefetch thread before returning (10-second timeout; verified both rows were cached). With the same [prefetch] case and scheduling constraint, the original test failed 5/5 runs with empty captured warnings; the fixed test passed 5/5. All three modes also passed in five probe runs. This verifies the ordering mechanism, not the natural flake rate; the probe is not included in the patch.

No visual change; screenshots are not applicable.

@SigureMo Please review when convenient.

# ✅ Tests

- Move shuffle after the warning monkeypatch and inside try/finally so
  background prefetch cannot emit the missing-media warning before capture.
- Preserve the existing inline, async and prefetch assertions; leave
  production behavior unchanged.
@rai-studio-bot

rai-studio-bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Nyanpasu 审查看板

审查状态: ✅ 已通过

审查版本: 87f4ce9b0e6462fdb8067b9433bf70936cb68249

已完成审查,未发现需修改的问题;警告捕获已先于预取启动,三种模式及断言保留。本地测试受环境限制未完成;Pre-commit CI 已通过,其余 CI 尚在运行。

没有未解决的审查问题。

Powered by Nyanpasu with gpt-6-astra medium, please check the suggestions carefully.

@rai-studio-bot rai-studio-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已审查 87f4ce9b,未发现需要修改的问题。捕获回调现在先于预取线程启动,原有三种模式与断言保持不变,且启动过程纳入了 finally 清理。

本地测试未完成:默认环境缺少 PyTorch,另一现有环境启动 pytest 时退出码为 139。Pre-commit CI 已通过,其余 CI 尚在运行;本次未运行 GPU/多节点集成测试(无对应测试集群)。

Powered by Nyanpasu with gpt-6-astra medium, please check the suggestions carefully.

@SigureMo SigureMo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTMeow 🐾

15 次 rerun 均通过,稳定性应该没问题了

@SigureMo
SigureMo merged commit ffa074d into redai-studio:main Sep 23, 2026
67 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants