feat(detect): index Jupyter notebooks via markdown sidecars

[KunojiLym]

Summary

• Convert .ipynb files to Markdown sidecars (code cells as fenced blocks using the kernel language from notebook metadata, markdown cells verbatim, outputs stripped) and classify them as documents so notebook-heavy corpora are no longer silently dropped during scan
• Sidecar names use the scan-root-relative path (Office sidecar name is keyed on the absolute source path, so every checkout of the same repo emits a duplicate converted/*.md #2059); re-runs that only change outputs do not bump the sidecar or trigger re-extraction
• Document support in README, how-it-works, and CHANGELOG; no new dependencies

Fixes #1497

Testing

• uv run pytest tests/test_detect.py -k "ipynb or notebook or sidecar or convert_office" -q
• uv run pytest tests/test_office_incremental.py tests/test_office_limits.py -q (office path unchanged / non-regression)
• Offline checks: convert_office_file byte-identical to base; detect/detect_incremental notebook path (sidecar create, source refresh, output-only unchanged)
• Maintainer: approve fork CI workflow run if Actions still shows action_required

[graphify-labs]

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

---

Graphify review — findings

This PR adds Jupyter notebook (.ipynb) support to the file detection pipeline. It introduces NOTEBOOK_EXTENSIONS, classifies notebooks as DOCUMENT, and adds two functions—ipynb_to_markdown (converts a notebook to markdown, stripping outputs and using the kernel language for code fences) and convert_notebook_file (writes a hash-named markdown sidecar, mirroring the existing office-file conversion). The detect flow is updated to convert notebooks to sidecars during scanning, and a large block of tests covers classification, conversion, sidecar rewrite/preservation behavior on source-vs-output changes, and incremental detection. The surface area touched is graphify/detect.py (constants, classify_file, new conversion helpers, the detect loop) and tests/test_detect.py (new notebook-related test cases and helpers).

Worth a look

• convert_notebook_file uses path.resolve() while convert_office_file uses root-relative naming, breaking stability across checkouts — graphify/detect.py:815 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
• Notebook sidecar name uses absolute resolved path, not scan-root-relative — unstable across checkouts — graphify/detect.py:833 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1163 functions depend on the 436 functions this change touches.

Health — this change adds coupling hotspots:

• worse: detect() — 83 callers, 13 callees
• worse: classify_file() — 43 callers, 3 callees
• worse: detect_incremental() — 16 callers, 5 callees
• worse: count_words() — 3 callers, 4 callees

Verification — 1163 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 629 function(s) in the blast radius were not formally verified this run

· 1 grounded finding(s) anchored inline below; 3 more finding(s) on lines outside this diff (see the check run).

[graphify-labs]

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

---

Graphify review — findings

This PR adds Jupyter notebook (.ipynb) support to the file detection/conversion pipeline in graphify/detect.py. It introduces a NOTEBOOK_EXTENSIONS set, classifies notebooks as DOCUMENT, and adds two functions—ipynb_to_markdown (extracts cell sources into fenced markdown, stripping outputs and using kernel language for fences) and convert_notebook_file (writes a hashed markdown sidecar, comparing extracted content rather than mtime to decide whether to rewrite). The detect flow is wired to convert notebooks into sidecars similar to the existing Office-file handling. The bulk of the change is new tests in tests/test_detect.py covering classification, markdown conversion across cell types/kernel languages/malformed inputs, sidecar creation, and sidecar rewrite/mtime behavior on notebook re-runs.

No blocking issues surfaced. 7 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1176 functions depend on the 449 functions this change touches.

Health — this change adds coupling hotspots:

• worse: detect() — 83 callers, 12 callees
• worse: classify_file() — 43 callers, 3 callees
• worse: detect_incremental() — 16 callers, 5 callees
• new: convert_notebook_file() — 7 callers, 2 callees

Verification — 1176 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 642 function(s) in the blast radius were not formally verified this run

· 1 grounded finding(s) anchored inline below; 3 more finding(s) on lines outside this diff (see the check run).

[graphify-labs]

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

---

Graphify review — findings

This PR adds Jupyter notebook (.ipynb) support to the file detection/conversion pipeline in graphify/detect.py. It introduces a NOTEBOOK_EXTENSIONS set, classifies notebooks as DOCUMENT, and adds ipynb_to_markdown and convert_notebook_file functions that convert notebooks into markdown sidecars (stripping outputs, using kernel language for code fences). A shared _sidecar_path helper is extracted from the existing Office conversion path so both Office and notebook conversions use the same relative-path-based, NFC-normalized naming scheme. The detect function is wired to convert notebooks to sidecars during scanning, mirroring the existing Office-file handling. The test file adds coverage for notebook classification, markdown conversion across cell types/kernel languages/malformed input, and end-to-end sidecar creation during detection. The surface area touches file classification, the conversion/sidecar helpers, and the main detect loop.

No blocking issues surfaced. 9 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1172 functions depend on the 445 functions this change touches.

Health — this change adds coupling hotspots:

• worse: detect() — 83 callers, 12 callees
• worse: classify_file() — 43 callers, 3 callees
• worse: detect_incremental() — 16 callers, 5 callees
• worse: convert_office_file() — 8 callers, 4 callees

Verification — 1172 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 638 function(s) in the blast radius were not formally verified this run

· 1 grounded finding(s) anchored inline below; 3 more finding(s) on lines outside this diff (see the check run).

[graphify-labs]

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

---

Graphify review — findings

This PR adds Jupyter notebook (.ipynb) support to the file detection pipeline. In graphify/detect.py, it introduces a NOTEBOOK_EXTENSIONS set, classifies notebooks as DOCUMENT, and adds helper functions (ipynb_to_markdown, convert_notebook_file, _notebook_sidecar_path) that convert notebooks into markdown sidecar files—stripping outputs, using the kernel language for code fences, and mirroring the Office sidecar naming scheme. The detect() flow is wired to convert notebooks into sidecars rather than indexing the raw .ipynb. The test file adds coverage for notebook classification, markdown conversion across various cell/metadata/source shapes, sidecar naming stability across checkouts, and rewrite/refresh semantics on re-runs versus source edits. The surface area is the detection module and its test suite; the many rationale_* symbol changes suggest accompanying doc/rationale entries were touched as well.

No blocking issues surfaced. 4 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1172 functions depend on the 445 functions this change touches.

Health — this change adds coupling hotspots:

• worse: detect() — 83 callers, 12 callees
• worse: classify_file() — 43 callers, 3 callees
• worse: detect_incremental() — 16 callers, 5 callees

Verification — 1172 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 638 function(s) in the blast radius were not formally verified this run

· 3 more finding(s) on lines outside this diff (see the check run).

[graphify-labs]

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

---

Graphify review — findings

This PR adds support for indexing Jupyter notebooks (.ipynb) by converting them into Markdown sidecars before Pass 3 extraction, rather than treating them as code or doc files directly. In detect.py it introduces a NOTEBOOK_EXTENSIONS set, classifies notebooks as DOCUMENT, and adds ipynb_to_markdown, convert_notebook_file, and _notebook_sidecar_path helpers that emit fenced code (using the kernel language from metadata) and verbatim markdown while stripping outputs, wired into the main detect loop alongside the existing Office conversion path. Supporting changes include new tests in test_detect.py covering classification, conversion, and sidecar behavior, plus CHANGELOG, README, and how-it-works documentation updates.

No blocking issues surfaced. 4 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1382 functions depend on the 655 functions this change touches.

Health — this change adds coupling hotspots:

• worse: detect() — 83 callers, 12 callees
• worse: classify_file() — 43 callers, 3 callees
• worse: detect_incremental() — 16 callers, 5 callees

Verification — 1382 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 848 function(s) in the blast radius were not formally verified this run

· 3 more finding(s) on lines outside this diff (see the check run).

[graphify-labs]

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

---

Graphify review — findings

This PR adds support for indexing Jupyter notebooks (.ipynb) by converting them into Markdown sidecars, mirroring the existing Office-file conversion flow. It introduces NOTEBOOK_EXTENSIONS, classifies notebooks as DOCUMENT, and adds ipynb_to_markdown, convert_notebook_file, and _notebook_sidecar_path helpers that extract cell sources (code cells fenced with the kernel language, markdown verbatim, outputs stripped) and write sidecars using a scan-root-relative hashed name. The detect loop is wired to convert notebooks, and the changes are accompanied by new tests, README/docs updates, and a changelog entry. Surface area: graphify/detect.py (constants, classification, new conversion functions, detect loop), tests/test_detect.py, README.md, docs/how-it-works.md, and CHANGELOG.md.

No blocking issues surfaced. 4 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1403 functions depend on the 656 functions this change touches.

Health — this change adds coupling hotspots:

• worse: detect() — 83 callers, 12 callees
• worse: classify_file() — 43 callers, 3 callees
• worse: detect_incremental() — 16 callers, 5 callees

Verification — 1403 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 849 function(s) in the blast radius were not formally verified this run

· 3 more finding(s) on lines outside this diff (see the check run).

[graphify-labs]

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

---

Graphify review — findings

This pull request adds support for Jupyter notebook (.ipynb) files as a new document type. It introduces conversion logic in graphify/detect.py that transforms notebooks into Markdown sidecars (code cells become fenced blocks using the kernel language, markdown cells pass through verbatim, outputs stripped), following the same hash-based sidecar naming scheme used for Office files but using content comparison rather than mtime for the rewrite check. Notebooks are classified as DOCUMENT, wired into the detect pipeline, documented in the README and how-it-works docs, and covered by new tests exercising cell conversion, language fencing, malformed input, sidecar creation, and incremental behavior.

No blocking issues surfaced. 5 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1275 functions depend on the 491 functions this change touches.

Health — this change adds coupling hotspots:

• worse: detect() — 85 callers, 12 callees
• worse: classify_file() — 43 callers, 3 callees
• worse: detect_incremental() — 18 callers, 5 callees

Verification — 1275 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 696 function(s) in the blast radius were not formally verified this run

· 3 more finding(s) on lines outside this diff (see the check run).