Add PyLucene integration and CPU/GPU end-to-end tests

.gitignore

@@ -7,3 +7,5 @@ target
 bin
 .project
 cuvs-workdir
+__pycache__/
+.pytest_cache/

README.md

@@ -104,6 +104,147 @@ public class HelloCuvsLucene {
 
 The artifacts would be built and available in the target / folder.
 
+### Using with PyLucene
+
+PyLucene embeds a JVM and starts it with the classpath passed to `lucene.initVM(...)`.
+Because PyLucene's generated Python module only exposes the Java classes it was built
+to wrap, use Lucene's service provider lookup to load `cuvs-lucene` codecs from
+Python instead of importing `com.nvidia.cuvs.lucene` classes directly.
+
+Build the standard cuvs-lucene jar:
+
+```sh
+mvn clean package -DskipTests
+```
+
+Then start PyLucene with the base `cuvs-java` jar, the standard `cuvs-lucene`
+jar, and PyLucene's own Lucene classpath:
+
+```python
+import os
+from pathlib import Path
+
+import lucene
+
+cuvs_java_jar = Path(os.environ["CUVS_LUCENE_CUVS_JAVA_JAR"])
+cuvs_lucene_jar = next(
+    jar
+    for jar in Path("target").glob("cuvs-lucene-*.jar")
+    if "-jar-with-" not in jar.name
+    and not jar.name.endswith(("-sources.jar", "-javadoc.jar"))
+)
+lucene.initVM(
+    classpath=os.pathsep.join(
+        [str(cuvs_java_jar), str(cuvs_lucene_jar), lucene.CLASSPATH]
+    ),
+    vmargs=[
+        "--enable-native-access=ALL-UNNAMED",
+        "--add-modules=jdk.incubator.vector",
+    ],
+)
+
+from org.apache.lucene.codecs import Codec
+
+codec = Codec.forName("Lucene101AcceleratedHNSWCodec")
+```
+
+Use the returned `codec` with `IndexWriterConfig.setCodec(codec)`. The standard
+artifact includes `cuvs-lucene` classes and service descriptors.
+PyLucene must provide Lucene classes, and the base multi-release `cuvs-java` jar
+must be present separately on the JVM classpath. Do not use a native classifier
+`cuvs-java` jar here unless you also want to rely on its embedded native
+libraries; the base jar uses native libraries from
+`LD_LIBRARY_PATH`/`java.library.path`.
+
+#### PyLucene end-to-end tests
+
+Pytest cases are under `src/test/python`. The parametrized cases and assertions
+are in `test_pylucene_end_to_end.py`; reusable index and search code is in
+`pylucene_test_support.py`. The test-only Java adapters in
+`src/test/java/com/nvidia/cuvs/lucene/PyLuceneTestSupport.java` are compiled to
+`target/test-classes` and are not included in the published jar.
+
+`ci/run_pylucene_pytests.sh` builds the Maven artifacts when requested, resolves
+the PyLucene classpath inputs, and invokes pytest. `test_pylucene.sh` at the
+repository root is the convenient entry point.
+
+To build the artifacts and run the default CPU check in an activated PyLucene
+environment:
+
+```sh
+./test_pylucene.sh
+```
+
+The default runs the jar-packaging checks and `cpu-hnsw-1-segment`. Use the
+groups below for broader coverage.
+Use `--no-build` when the standard jar and test bridge are already compiled.
+`CUVS_LUCENE_PYLUCENE_TEST_CLASSES` can point to a different test-classes
+directory and defaults to `target/test-classes`.
+
+Pytest can also be invoked directly once the PyLucene environment and classpath
+inputs are available:
+
+```sh
+CUVS_LUCENE_JAR=/absolute/path/to/cuvs-lucene.jar \
+CUVS_LUCENE_CUVS_JAVA_JAR=/absolute/path/to/cuvs-java.jar \
+CUVS_LUCENE_PYLUCENE_TEST_CLASSES="$(pwd)/target/test-classes" \
+python3 -m pytest -q -s src/test/python/test_pylucene_end_to_end.py
+```
+
+The execution-path groups are:
+
+- `cpu-hnsw`: HNSW build and search through Lucene's CPU path.
+- `gpu-cagra-built-hnsw`: GPU CAGRA build followed by HNSW search.
+- `gpu-cagra-search`: GPU CAGRA build and search.
+
+GPU cases assert that cuVS was actually used and fail if it is unavailable or
+falls back to CPU. CPU cases assert and report the CPU path, and HNSW cases
+verify the persisted graph shape. CAGRA cases use `graphDegree=32` and
+`intermediateGraphDegree=64`.
+
+Vectors and queries are deterministic. Expected neighbors are computed with
+brute force. Queries for live indexed vectors check rank-one self matches,
+duplicate hits, and a configurable recall floor. Separate tests cover segment
+topology, force merges, HNSW layer count, CAGRA `searchWidth` values 1, 16, and
+32, and document filters. A dedicated CAGRA-search case verifies that a deleted
+document is not returned. Set the recall floor with `--min-recall=FLOAT` in the
+wrapper or `CUVS_LUCENE_PYLUCENE_MIN_RECALL` for direct pytest execution.
+The default floor is `0.75`.
+
+The `document-filter` group exercises selective filters through CPU HNSW,
+CAGRA-built HNSW, and CAGRA search. The CAGRA-search case uses ten segments
+and accepts roughly one quarter of each segment. Every segment retains more
+than `topK` accepted vectors so Lucene exercises approximate native
+prefiltering; results are checked against brute-force neighbors from only the
+accepted vectors.
+
+Useful behavior groups include `execution-paths`, `segment-topologies`,
+`force-merges`, `hnsw-layer-counts`, `cagra-search-widths`,
+`deleted-documents`, and `document-filter`.
+
+Run the complete CPU/GPU end-to-end suite with:
+
+```sh
+./test_pylucene.sh --full-e2e
+```
+
+Select a focused group or set the minimum document count per scenario:
+
+```sh
+./test_pylucene.sh --cases=cagra-search-widths \
+  --rows=5000 --dims=64 --topk=20 --min-recall=0.8
+```
+
+`--rows` is a lower bound. CAGRA cases use at least 97 vector-bearing
+documents per constructed graph, one more than cuVS's internal NN-Descent
+degree of 96. The three-layer HNSW case uses at least 24,832 vector-bearing
+documents so its third layer retains 97. Arguments after `--` are passed to
+pytest, for example:
+
+```sh
+./test_pylucene.sh --no-build --cases=gpu-cagra-search -- -x
+```
+
 ### Running Tests
 
 ```sh

ci/run_pylucene_pytests.sh

@@ -0,0 +1,232 @@
+#!/bin/bash
+
+# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "${REPO_ROOT}"
+
+MVN_BIN="${MVN:-mvn}"
+PYTHON_BIN="${PYTHON:-python3}"
+SKIP_BUILD=0
+FULL_END_TO_END=0
+PYLUCENE_CASES="${CUVS_LUCENE_PYLUCENE_CASES:-}"
+PYLUCENE_ROWS="${CUVS_LUCENE_PYLUCENE_ROWS:-}"
+PYLUCENE_DIMS="${CUVS_LUCENE_PYLUCENE_DIMS:-}"
+PYLUCENE_TOPK="${CUVS_LUCENE_PYLUCENE_TOPK:-}"
+PYLUCENE_MIN_RECALL="${CUVS_LUCENE_PYLUCENE_MIN_RECALL:-}"
+CUVS_LUCENE_JAR_PATH="${CUVS_LUCENE_JAR:-}"
+PYLUCENE_TEST_CLASSES_PATH="${CUVS_LUCENE_PYLUCENE_TEST_CLASSES:-target/test-classes}"
+PYTEST_ARGS=()
+
+while [[ "$#" -gt 0 ]]; do
+  arg="$1"
+  shift
+  case "${arg}" in
+    --full-e2e|--gpu-e2e)
+      FULL_END_TO_END=1
+      ;;
+    --no-build)
+      SKIP_BUILD=1
+      ;;
+    --cases=*)
+      PYLUCENE_CASES="${arg#--cases=}"
+      ;;
+    --rows=*)
+      PYLUCENE_ROWS="${arg#--rows=}"
+      ;;
+    --dims=*)
+      PYLUCENE_DIMS="${arg#--dims=}"
+      ;;
+    --topk=*)
+      PYLUCENE_TOPK="${arg#--topk=}"
+      ;;
+    --min-recall=*)
+      PYLUCENE_MIN_RECALL="${arg#--min-recall=}"
+      ;;
+    --)
+      PYTEST_ARGS+=("$@")
+      break
+      ;;
+    -h|--help)
+      echo "Usage: ./test_pylucene.sh [OPTIONS] [-- PYTEST_ARGS...]"
+      echo
+      echo "Prepares the PyLucene JVM, jar, classpath, and native-library inputs, then runs pytest."
+      echo
+      echo "Options:"
+      echo "  --no-build             Use existing Maven artifacts."
+      echo "  --full-e2e             Run the complete CPU/GPU end-to-end suite."
+      echo "  --cases=CASE[,CASE...] Select case names or groups."
+      echo "  --rows=N               Set the minimum document count per scenario."
+      echo "  --dims=N               Set vector dimensions."
+      echo "  --topk=N               Set requested neighbor count."
+      echo "  --min-recall=FLOAT     Set the brute-force recall floor (default: 0.75)."
+      echo "  -- PYTEST_ARGS         Forward remaining arguments to pytest."
+      echo
+      echo "Execution-path groups:"
+      echo "  cpu-hnsw, gpu-cagra-built-hnsw, gpu-cagra-search, gpu"
+      echo
+      echo "Behavior groups:"
+      echo "  execution-paths, segment-topologies, force-merges,"
+      echo "  hnsw-layer-counts, cagra-search-widths,"
+      echo "  deleted-documents, document-filter, all"
+      echo
+      echo "Environment overrides:"
+      echo "  CUVS_LUCENE_JAR, CUVS_LUCENE_CUVS_JAVA_JAR,"
+      echo "  CUVS_LUCENE_PYLUCENE_TEST_CLASSES,"
+      echo "  CUVS_LUCENE_PYLUCENE_CASES, CUVS_LUCENE_PYLUCENE_ROWS,"
+      echo "  CUVS_LUCENE_PYLUCENE_DIMS, CUVS_LUCENE_PYLUCENE_TOPK,"
+      echo "  CUVS_LUCENE_PYLUCENE_MIN_RECALL, PYTHON, MVN"
+      exit 0
+      ;;
+    *)
+      echo "Unknown argument: ${arg}" >&2
+      echo "Use -- before arguments intended for pytest." >&2
+      exit 2
+      ;;
+  esac
+done
+
+require_command() {
+  if ! command -v "$1" >/dev/null 2>&1; then
+    echo "Required command not found: $1" >&2
+    exit 127
+  fi
+}
+
+absolute_from_repo_root() {
+  case "$1" in
+    /*)
+      printf '%s\n' "$1"
+      ;;
+    *)
+      printf '%s/%s\n' "${REPO_ROOT}" "$1"
+      ;;
+  esac
+}
+
+find_cuvs_java_jar() {
+  if [[ -n "${CUVS_LUCENE_CUVS_JAVA_JAR:-}" ]]; then
+    printf '%s\n' "${CUVS_LUCENE_CUVS_JAVA_JAR}"
+    return
+  fi
+
+  local versioned_jar="${HOME}/.m2/repository/com/nvidia/cuvs/cuvs-java/${project_version}/cuvs-java-${project_version}.jar"
+  if [[ -f "${versioned_jar}" ]]; then
+    printf '%s\n' "${versioned_jar}"
+    return
+  fi
+
+  local m2_repo="${HOME}/.m2/repository/com/nvidia/cuvs/cuvs-java"
+  if [[ ! -d "${m2_repo}" ]]; then
+    return
+  fi
+
+  find "${m2_repo}" \
+    -type f \
+    -name 'cuvs-java-*.jar' \
+    ! -name '*sources*' \
+    ! -name '*javadoc*' \
+    ! -name '*x86_64*' \
+    | sort -V \
+    | tail -n 1
+}
+
+require_command "${PYTHON_BIN}"
+
+if [[ "${SKIP_BUILD}" -eq 0 && -z "${CUVS_LUCENE_JAR_PATH}" ]]; then
+  require_command "${MVN_BIN}"
+  "${MVN_BIN}" clean package -DskipTests
+fi
+
+project_version="$(
+  sed -n 's/.*CUVS_LUCENE#VERSION_UPDATE_MARKER_START--><version>\([^<]*\)<\/version>.*/\1/p' pom.xml \
+    | head -n 1
+)"
+if [[ -z "${project_version}" ]]; then
+  echo "Unable to determine project version from pom.xml" >&2
+  exit 1
+fi
+
+if [[ -n "${CUVS_LUCENE_JAR_PATH}" ]]; then
+  cuvs_lucene_jar="${CUVS_LUCENE_JAR_PATH}"
+else
+  cuvs_lucene_jar="target/cuvs-lucene-${project_version}.jar"
+fi
+cuvs_lucene_jar_abs="$(absolute_from_repo_root "${cuvs_lucene_jar}")"
+if [[ ! -f "${cuvs_lucene_jar_abs}" ]]; then
+  echo "cuvs-lucene jar not found: ${cuvs_lucene_jar_abs}" >&2
+  echo "Run without --no-build, or set CUVS_LUCENE_JAR." >&2
+  exit 1
+fi
+
+cuvs_java_jar="$(find_cuvs_java_jar)"
+if [[ -z "${cuvs_java_jar}" || ! -f "${cuvs_java_jar}" ]]; then
+  echo "Base cuvs-java jar not found." >&2
+  echo "Set CUVS_LUCENE_CUVS_JAVA_JAR to the base jar, not a native classifier jar." >&2
+  exit 1
+fi
+cuvs_java_jar_abs="$(absolute_from_repo_root "${cuvs_java_jar}")"
+
+pylucene_test_classes_abs="$(
+  absolute_from_repo_root "${PYLUCENE_TEST_CLASSES_PATH}"
+)"
+if [[ ! -d "${pylucene_test_classes_abs}" ]]; then
+  echo "PyLucene test classes not found: ${pylucene_test_classes_abs}" >&2
+  echo "Run Maven test compilation, or set CUVS_LUCENE_PYLUCENE_TEST_CLASSES." >&2
+  exit 1
+fi
+pylucene_test_classes_abs="$(
+  cd "${pylucene_test_classes_abs}" && pwd -P
+)"
+
+"${PYTHON_BIN}" -c "import lucene" >/dev/null 2>&1 || {
+  echo "Python cannot import PyLucene's lucene module." >&2
+  echo "Activate a PyLucene environment compatible with this project's Lucene version." >&2
+  exit 1
+}
+
+"${PYTHON_BIN}" -m pytest --version >/dev/null 2>&1 || {
+  echo "Python cannot run pytest. Install pytest in the active PyLucene environment." >&2
+  exit 1
+}
+
+pytest_env=(
+  "CUVS_LUCENE_JAR=${cuvs_lucene_jar_abs}"
+  "CUVS_LUCENE_CUVS_JAVA_JAR=${cuvs_java_jar_abs}"
+  "CUVS_LUCENE_PYLUCENE_TEST_CLASSES=${pylucene_test_classes_abs}"
+  "CUVS_LUCENE_VERIFY_ALL_CODECS=${CUVS_LUCENE_VERIFY_ALL_CODECS:-1}"
+)
+
+if [[ "${FULL_END_TO_END}" -eq 1 ]]; then
+  pytest_env+=(
+    "CUVS_LUCENE_PYLUCENE_CASES=${PYLUCENE_CASES:-all}"
+    "CUVS_LUCENE_PYLUCENE_ROWS=${PYLUCENE_ROWS:-2000}"
+    "CUVS_LUCENE_PYLUCENE_DIMS=${PYLUCENE_DIMS:-32}"
+    "CUVS_LUCENE_PYLUCENE_TOPK=${PYLUCENE_TOPK:-20}"
+  )
+else
+  if [[ -n "${PYLUCENE_CASES}" ]]; then
+    pytest_env+=("CUVS_LUCENE_PYLUCENE_CASES=${PYLUCENE_CASES}")
+  fi
+  if [[ -n "${PYLUCENE_ROWS}" ]]; then
+    pytest_env+=("CUVS_LUCENE_PYLUCENE_ROWS=${PYLUCENE_ROWS}")
+  fi
+  if [[ -n "${PYLUCENE_DIMS}" ]]; then
+    pytest_env+=("CUVS_LUCENE_PYLUCENE_DIMS=${PYLUCENE_DIMS}")
+  fi
+  if [[ -n "${PYLUCENE_TOPK}" ]]; then
+    pytest_env+=("CUVS_LUCENE_PYLUCENE_TOPK=${PYLUCENE_TOPK}")
+  fi
+fi
+
+if [[ -n "${PYLUCENE_MIN_RECALL}" ]]; then
+  pytest_env+=("CUVS_LUCENE_PYLUCENE_MIN_RECALL=${PYLUCENE_MIN_RECALL}")
+fi
+
+env "${pytest_env[@]}" \
+  "${PYTHON_BIN}" -m pytest -q -s \
+  src/test/python/test_pylucene_end_to_end.py \
+  "${PYTEST_ARGS[@]}"

pom.xml

@@ -166,9 +166,9 @@
         <artifactId>maven-assembly-plugin</artifactId>
         <version>3.6.0</version>
         <configuration>
-          <descriptorRefs>
-            <descriptorRef>jar-with-dependencies</descriptorRef>
-          </descriptorRefs>
+          <descriptors>
+            <descriptor>src/main/assembly/jar-with-dependencies.xml</descriptor>
+          </descriptors>
         </configuration>
         <executions>
           <execution>