Add PyLucene integration and CPU/GPU end-to-end tests

[nvzm123]

Summary

This ports the cuVS-Lucene production changes from NVIDIA/cuvs-lucene#174 into java/cuvs-lucene after the project moved into the cuVS monorepo.

The port:

• targets the Lucene 10.2 runtime ABI while retaining the established Lucene101AcceleratedHNSWCodec SPI name;
• uses Lucene 10.2's Lucene101Codec as the accelerated codecs' explicit delegate;
• exposes correctly typed scalar and binary provider APIs while retaining legacy JVM descriptors with actionable failures;
• keeps providers version-scoped and rejects version-specific capabilities explicitly;
• initializes scalar- and binary-quantized vector formats lazily so SPI discovery cannot be poisoned by an unavailable optional Lucene format;
• preserves the concrete throwable type when reflective format construction fails;
• corrects GPU query-path documentation for multi-partition algorithm selection and global-document-ID result partitioning; and
• verifies the standard thin JAR's exact Lucene SPI descriptor/provider sets and checks that Lucene, base cuvs-java, and test-only PyLucene support are not bundled.

While reconciling the port with current main, the branch retained PR #2483's conditional padded-dataset selection for both aligned and unaligned CAGRA serialization. The writer and its aligned-dimension regression test have no net diff from current main.

The PyLucene pytest suite and its test-only Java adapters are owned by the cuVS Bench backend in NVIDIA/cuvs#2385, not by the production cuvs-lucene artifact. The standard thin cuvs-lucene JAR remains the artifact under test; Lucene and the base cuvs-java JAR remain external classpath dependencies.

Apache does not publish PyLucene 10.2.0, so end-to-end validation uses a custom PyLucene wrapper generated against the same Lucene 10.2.0 sources as this project.

Test coverage

The in-tree Java coverage adds or strengthens:

• fresh-JVM codec and vector-format SPI discovery;
• lazy scalar and binary format construction while observing an untouched, empty provider cache;
• per-version provider caching and actionable unsupported-capability failures;
• Lucene 10.2 binary-format capability and constructor-failure classification;
• legacy source and JVM-descriptor compatibility;
• explicit CPU-fallback and GPU query-routing boundaries; and
• exact thin-JAR Lucene SPI and dependency/test-payload exclusion checks during Maven verify.

The dependent pytest matrix in #2385 proves these execution paths:

• CPU HNSW build and HNSW search;
• GPU CAGRA build followed by one-layer or three-layer HNSW search; and
• GPU CAGRA build and CAGRA search.

GPU-required cases assert the concrete accelerated writer, reader, and query implementations and fail on unavailable cuVS or CPU fallback. CPU cases explicitly verify a stock Lucene HNSW path.

The matrix covers a single live document, one and ten segments, 10-to-1 and 100-to-10 force merges, CAGRA searchWidth values 1, 16, and 32, deletions, vectorless documents, selective filters, persisted HNSW graph degree/layers, exact filtered search, and deterministic brute-force recall. Assertions check rank-one self matches where applicable, exact hit counts, duplicate exclusion, inactive/filter-rejected document exclusion, and configurable recall floors. It also covers CAGRA-built HNSW search with top_k=2000 and num_candidates=2500; direct CAGRA search retains its supported k <= 1024 boundary.

Warning-free CAGRA cases use graphDegree=32, intermediateGraphDegree=64, and enough vectors to avoid graph-parameter clamping, including 24,832 vectors for the three-layer case.

Validation

Final revisions:

• cuVS producer PR Add PyLucene integration and CPU/GPU end-to-end tests #2475: 57ce4920374dce1cecd09091566b18710fe01c82
• cuVS Bench PR Add PyLucene backend for cuVS-Lucene vector search in cuvs-bench #2385: e54979c3b5aba2f53a77720f7ebc3dd0e16618fa

Validated on an NVIDIA A10G with JDK 22 and Lucene/PyLucene 10.2. The GPU run used the official RAPIDS 26.12 development nightly libcuvs 26.12.00a8 (cuda12_260910004918_29e1101b), built from the tested main revision 29e1101b merged into both branches. The runtime reported cuVS 26.12.0 and used the RMM 26.12 ABI; the cuvs-java and cuvs-lucene 26.12.0 JARs were rebuilt from the merged producer branch.

After that full run, PR #2475 merged current main (ef29c4cfd53082d31c1fc05ec35251c835120efa) in 57ce4920374dce1cecd09091566b18710fe01c82. That merge adds only the four upstream CMake dependency-discovery changes and leaves the validated Java, cuvs-lucene, PyLucene, and test inputs unchanged; git diff --check passed, and the GPU suite was not redundantly rerun.

Producer validation

• full GPU-enabled cuvs-java and cuvs-lucene Maven suites: 497 tests, 0 failures, 0 errors, 29 skipped (cuvs-java: 112 with 1 skipped; cuvs-lucene: 385 with 28 skipped, including both passing ThinJarContentsIT cases)

Dependent cuVS Bench validation

• python -m pytest -q -s cuvs_bench/tests/pylucene --run-pylucene: 393 passed in 55.05 seconds
• relevant shell syntax checks: passed

The Java suites exercised the native GPU paths and emitted no version-mismatch or linkage errors. Their inherited randomized and small-dataset cases emitted native cuVS graph-parameter diagnostics (271 warning lines from cuvs-java and 2,651 from cuvs-lucene). The PyLucene suite emitted no cuVS configuration or CPU-fallback warnings; it emitted the expected JVM notice for the incubating vector module.

Baseline randomized-test failure observed during validation

An unseeded full Maven run selected seed CC0EA94328BAB3E5 and failed in TestCuVSVectorsFormat.testRandomWithUpdatesAndGraph with IllegalStateException: Index not found for field:field. This exact seed and failure signature were first recorded during NVIDIA/cuvs-lucene#174 validation on August 17, 2026; no earlier public issue tracking this exact failure was found.

The first commit in this port, 435c52ec, has the untouched cuVS-Lucene import 0fa5ebe5 as its direct parent. That baseline imported the standalone cuVS-Lucene source from 61431aa, which GitHub records as #174's base commit; #174 was closed without merging when the project moved. A clean checkout of 0fa5ebe5, which contains none of the port delta, reproduces the same primary and secondary seeds (CC0EA94328BAB3E5:B2D36C5E9FD11B86), exception, and stack trace. This establishes that the failure exists independently of the changes ported from #174; it does not claim that its root cause was already tracked upstream. Historical reproduction command from java/cuvs-lucene:

mvn -q -Dtest=TestCuVSVectorsFormat#testRandomWithUpdatesAndGraph \
  -Dtests.seed=CC0EA94328BAB3E5 \
  -Dtests.locale=ti-Ethi-ET \
  -Dtests.timezone=Greenwich test

During the same #174 validation, a separate full run pinned to seed 5A17C10120260817 passed; no multi-seed sweep was run. The exact 26.12 GPU-enabled Maven run at the current pushed head executed and passed TestCuVSVectorsFormat.testRandomWithUpdatesAndGraph. It did not rerun the historical failing seed, so this successful randomized invocation does not clear that seed-specific baseline failure.

Follow-up multithreaded concurrency coverage is tracked in NVIDIA/cuvs#2407.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cjnolet]

This still needs some work.