Skip to content

fix(ci): collect the nooa-memory test suite 🤖🤖🤖 - #292

Open
sushant-mishra-dtu wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
sushant-mishra-dtu:fix/collect-nooa-memory-tests
Open

fix(ci): collect the nooa-memory test suite 🤖🤖🤖#292
sushant-mishra-dtu wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
sushant-mishra-dtu:fix/collect-nooa-memory-tests

Conversation

@sushant-mishra-dtu

@sushant-mishra-dtu sushant-mishra-dtu commented Sep 5, 2026

Copy link
Copy Markdown

What is wrong

testpaths in pyproject.toml lists three of the four packages under packages/
nooa-cli, nooa-acp, nooa-bench — but not nooa-memory. The CI unit-test step
(.github/workflows/ci.yml:46) runs

uv run pytest -q -m "not integration and not stress and not sandbox"

with no path arguments, so pytest applies testpaths and never reaches
packages/nooa-memory/tests. The suite is installed and importable in CI —
uv sync --all-extras --no-extra sandbox pulls it in via the memory extra
(pyproject.toml:88) and it is a declared workspace member (:72) — it is simply
never collected.

Net effect: a shipped optional package has no regression protection in CI. Nothing
in nooa-memory can fail the build.

Evidence

# CI's exact invocation, collect-only, on main
$ uv run pytest --collect-only -q -m "not integration and not stress and not sandbox"
7018/7262 tests collected
$ ... | grep -c nooa-memory
0

# the same tests, addressed directly
$ uv run pytest packages/nooa-memory/tests -q
282 collected

With this one-line change, the same CI-shaped collection goes 7018 → 7300, exactly
the 282 that were dark.

Why this is worth doing now

There are four open PRs against nooa-memory right now — #277, #278, #279 and #284
and each one adds or edits a file under packages/nooa-memory/tests/. Those
regression tests cannot currently fail, on those PRs or on any later change that breaks
them.

This is also a failure mode the project already treats as a bug. ci.yml:40-44 installs
ripgrep specifically because otherwise "those tests silently skip and the job passes
vacuously. See #65", and the release and gitleaks steps (ci.yml:190, :234) both
hard-fail rather than pass vacuously. This is the same policy applied to a directory the
config never points at.

What I could and could not verify

I ran the suite on Windows, so my numbers are not a substitute for your runner:

uv run pytest packages/nooa-memory/tests -q
  -> 266 passed, 3 failed, 13 skipped

PYTHONUTF8=1 uv run pytest packages/nooa-memory/tests -q
  -> 268 passed, 1 failed, 13 skipped

All three failures look platform-specific rather than product defects:

  • two are in tests/memory/test_memory_readme.py, which calls Path.read_text() with no
    encoding= and so fails on a cp1252 console — they pass under PYTHONUTF8=1, and would
    pass on a UTF-8 runner;
  • test_file_capture_rejects_escaping_paths expects file:/etc/passwd to take the
    "must be relative" branch, but pathlib.Path("/etc/passwd").is_absolute() is False on
    Windows, so it takes the "escapes the working dir" branch instead. The path is still
    rejected; only the message differs.

The full suite on this branch adds exactly those tests and moves nothing else. Both runs
behind the same harness, both under PYTHONUTF8=1 so they are comparable to each other:

Tree failed passed skipped errors
main @ e137e1b 46 6,691 159 61
this branch 47 6,959 172 61

The delta — +1 failed, +268 passed, +13 skipped, errors unchanged — is exactly the
nooa-memory suite's standalone result above, so the one-line change adds that suite and
touches nothing else. The absolute failure count is high only because this is Windows,
where a lot of the suite is known-broken (#84, #95, #110); the delta is the point, not the
level.

So I expect all 282 to be green on ubuntu-latest, but I have not been able to prove
that.
If turning them on does surface something, say so and I will fix the fallout in
this PR rather than leave you with a red build.

Scope

One line. No source changes, no behaviour changes, no agent-facing contract touched.

Deliberately not included: util/eval_pipeline/tests is the other test directory
outside testpaths. It is dev tooling under util/ rather than a shipped package, and I
have not audited its state, so it seemed wrong to bundle it into a one-line fix. Happy to
look at it separately if you want it collected too.

Summary by CodeRabbit

  • Tests
    • Updated test discovery configuration to include the memory package’s test suite.

`testpaths` names packages/nooa-cli/tests, packages/nooa-acp/tests and
packages/nooa-bench/tests, but not packages/nooa-memory/tests. The CI unit-test
step runs pytest with no path arguments, so testpaths applies and the directory
is never reached — a shipped optional package with no regression protection.

The package is installed in CI (`uv sync --all-extras --no-extra sandbox` pulls
it in via the `memory` extra) and is a declared workspace member; only the
collection path was missing.

Measured with CI's exact invocation:

    pytest --collect-only -q -m "not integration and not stress and not sandbox"
    before: 7018 collected, 0 from nooa-memory
    after : 7300 collected  (+282)

Same shape as the ripgrep step added in NVIDIA-NeMo#65, which exists because otherwise
"those tests silently skip and the job passes vacuously".

Signed-off-by: sushant-mishra-dtu <sushant.arh@gmail.com>
@coderabbitai

coderabbitai Bot commented Sep 5, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a96b11b4-ffd6-420e-9133-879053957761

📥 Commits

Reviewing files that changed from the base of the PR and between e137e1b and c814754.

📒 Files selected for processing (1)
  • pyproject.toml

Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The pytest configuration now includes packages/nooa-memory/tests in the test paths.

Changes

Pytest configuration

Layer / File(s) Summary
Include nooa-memory tests
pyproject.toml
The pytest test paths now include the nooa-memory package tests.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to c8147

This change expands the default pytest run to include nooa-memory coverage, with no remaining concrete merge-blocking risk.

Suggested reviewers: sklinglernv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: updating CI test collection to include the nooa-memory test suite.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant