Skip to content

Add ImageGather.jl as a second RAG corpus, portability fixes, and unit tests#1

Open
flexie wants to merge 4 commits into
mainfrom
felix
Open

Add ImageGather.jl as a second RAG corpus, portability fixes, and unit tests#1
flexie wants to merge 4 commits into
mainfrom
felix

Conversation

@flexie

@flexie flexie commented Jun 10, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds ImageGather.jl as a second RAG corpus alongside JUDI.jl. The agent can now ground generated Julia in ImageGather workflows (surface and subsurface common-image gathers, extended Born modeling) as well as JUDI.
  • The corpus lives at src/judiagent/rag/imagegather/, cloned from slimgroup/ImageGather.jl@ext_PhyParams with the upstream LICENSE preserved. Provenance is recorded in src/judiagent/rag/THIRD_PARTY.md.
  • New retrieval tool search_imagegather_examples is built via the existing _build_retrieval_tool factory and wired into both the iterative and the autonomous ReAct agents. search_codebase now greps both rag/judi/ and rag/imagegather/ in a single invocation.
  • The three ImageGather example scripts received JUDI-style # # Heading markers so the Julia example chunker produces ~19 well-titled chunks instead of three whole-file blobs. Code itself is unchanged.
  • Portability fix in env/{common,desktop,pace}-local.sh: resolve script path via ${BASH_SOURCE[0]:-$0} and short-circuit the executed-not-sourced guard when ZSH_VERSION is set, so the helpers can be sourced under zsh on macOS. Verified live that bash behavior is unchanged.
  • Dropped the bogus "Programming Language :: Julia" PyPI classifier from pyproject.toml (not on the trove list; trips twine check).
  • 16 new unit tests under tests/unit_tests/test_imagegather_rag.py cover catalog wiring, tool surface, codebase-grep widening, chunked-output shape, agent tool binding, and provenance. Full unit suite goes from 38 → 54, all green.

Commits

  • f62e869 — Add ImageGather.jl as a second RAG corpus
  • a4d754f — Make env helpers source-able from zsh and drop bogus PyPI classifier
  • 5cd3cd4 — Add unit tests for ImageGather.jl RAG integration

Test plan

  • uv run pytest tests/unit_tests — 54 passed
  • Verified end-to-end pipeline: cache load → markdown/Julia chunking → 19 imagegather example chunks with named headings, 5 imagegather doc chunks
  • Verified search_codebase returns hits from /rag/imagegather/ for imagegather-only terms and still from /rag/judi/ for JUDI-only terms
  • Verified env/desktop-local.sh and env/pace-local.sh source cleanly under bash 5.3 and still refuse to run when executed
  • Reviewer to run an agent query targeting ImageGather (e.g. "show me how ImageGather.jl forms an extended common-image gather with physical parameters") and confirm the retriever creates rag/loaded_store/loaded_imagegather_examples.pkl and rag/retriever_store/retriever_imagegather_examples_<embedding_family>/ on first invocation
  • Reviewer to delete any pre-existing per-source caches before first run (instructions in the new CHANGELOG.md)

Notes for reviewers

  • The imagegather/src/ and imagegather/test/ directories are included verbatim from upstream for license/citation completeness but are not indexed (catalog filters to .md under docs/src/ and .jl under examples/).
  • Agent prompt (prompting/shared.py::GATHER_INTELLIGENCE) now mentions both tools so the model can pick the right corpus for a query.

fherrmann9 and others added 3 commits June 10, 2026 22:09
Clone slimgroup/ImageGather.jl@ext_PhyParams into src/judiagent/rag/imagegather/
and wire it through the existing source-agnostic RAG plumbing: a new RAG_CATALOG
entry (docs + examples), a search_imagegather_examples retrieval tool, and tool
list updates for both the iterative and ReAct agents. search_codebase now greps
both judi/ and imagegather/. The ImageGather example scripts get JUDI-style
`# # Heading` markers so the Julia example chunker yields ~19 well-titled
chunks instead of three whole-file blobs.

Provenance for the new corpus is recorded in src/judiagent/rag/THIRD_PARTY.md;
the CHANGELOG captures the operator step of clearing per-source caches before
the next agent run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sixteen tests covering everything we wired in for the new corpus:

- Catalog: imagegather is registered with both docs and examples entries,
  and its cache files / collection names / persist dirs are disjoint from
  JUDI's so the two indices cannot collide.
- Source layout: imagegather/docs/src and imagegather/examples exist and
  contain files of the declared filetypes.
- Tool surface: search_imagegather_examples is exported from
  judiagent.tools, has the expected name, and references the package in
  its description.
- Codebase grep: search_codebase finds imagegather-only terms (proving the
  second search root was added) and still finds JUDI-only terms
  (regression guard for the widening).
- Chunking: the JUDI-style `# # Heading` markers we added to the three
  example scripts produce multiple named chunks per file (instead of the
  prior one-chunk-per-file behavior), every chunk carries a non-empty
  heading, and the docs corpus chunks too.
- Agent binding: both iterative_agent and react_agent register the new
  tool, and GATHER_INTELLIGENCE mentions it.
- Provenance: the upstream LICENSE is present and THIRD_PARTY.md lists
  ImageGather.jl on branch ext_PhyParams.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…clean

Trim src/judiagent/rag/imagegather/ to the material actually used by RAG and
attribution — drop the upstream Julia source (src/), tests (test/), CI configs
(.github/), Project.toml manifests, docs build script (docs/make.jl), bib
file, figures (docs/img/), and the example notebook. Keep LICENSE, README.md
(which links to the upstream repo for the dropped artifacts), docs/src/*.md,
and examples/*.jl. THIRD_PARTY.md and CHANGELOG.md updated to describe the
trimmed snapshot accurately.

Refactor the two chunking tests to use pytest tmp_path so the pickle cache
no longer lands under src/judiagent/rag/loaded_store/. A small helper
_spec_with_tmp_cache uses dataclasses.replace to override the spec's
cache_file path while leaving every other field intact. Verified post-run
that the loaded_store/ tree contains no imagegather pickles produced by the
suite.

Ruff-formatted import block in tests/unit_tests/test_imagegather_rag.py
(ruff check src tests now passes). git diff --check is clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This classifier is not registered on PyPI's trove list, so hatchling's
metadata.validate_fields() raises ValueError during build_editable, which
in turn makes `uv sync` / `uv run` fail with:

    Failed to build judiagent ...
    Unknown classifier in field `project.classifiers`:
        Programming Language :: Julia

Removing the single offending line unblocks editable installs. The "Julia"
keyword on line 13 is untouched — project.keywords is free-form, only
classifiers are validated against the trove list.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant