Optimize CAGRA HNSW FBIN bulk indexing

[nvzm123]

Summary

This extends the controlled CagraHnswBulkIndexWriter API added in #2481 (and since reverted in #2597) with opt-in mapped and immutable external-FBIN ingestion for CAGRA-to-HNSW index builds. External-file ownership and one-shot lifecycle controls remain outside the generic Lucene codec API.

Related to cuvs-issue #2592.

Motivation

Bulk indexing often begins with a complete FBIN on local disk. Normal Lucene ingestion decodes and copies every vector into an intermediate native matrix, then persists another complete vector copy in the Lucene index. This is particularly expensive for high-dimensional datasets.

The bulk writer owns the IndexWriter configuration and lifecycle, allowing it to enforce the conditions borrowed storage requires: an exact dense-vector count and order, one controlled flush per segment, no index sort, and no vector merges.

Changes

• Adds mapped, self-contained FBIN indexing. CAGRA and the flat-vector writer share a read-only mapping during construction; the completed index retains its own vectors and can be moved independently.
• Adds immutable external-FBIN indexing. Each segment stores a checksummed, content-addressed descriptor instead of duplicating the .vec payload.
• Adds a classloader-local, reference-counted FBIN registry with structural, range, checksum, relocation, and reader-lifecycle validation.
• Shares one owning full-file mapping across live readers of the same external artifact, while preserving independent per-segment payload slices.
• Supports an exact number of contiguous, independently built segments. Segment indexes are combined through addIndexes and HardlinkCopyDirectoryWrapper without merging their graphs.
• Preserves absolute source IDs and Lucene document ordering across segments.
• Adds bounded host-side preparation overlap while conservatively serializing each segment's commit/close region, including GPU construction.
• Makes failed explicit commits non-retryable during close, uses collision-free temporary build directories, cancels and joins failed workers, and validates Lucene document-count limits before allocating or publishing.
• Validates independently encoded descriptor-v0 fixtures and converts wrong-source shape/range/overflow failures into contextual checked exceptions.
• Reports cumulative build-stage and counter metrics, selected parameters, returned base-graph
  degree, and actual adjacency bytes with truthful CPU/GPU/PCIe/disk labels and min/max gauges when
  segments differ.
• Adds fail-closed GPU CI coverage for mapped/external multi-segment builds and a real three-layer accelerated graph round trip.
• Tightens native-resource cleanup, source/target identity, exact segment-count, source-immutability, and registration-lease contracts.
• Keeps unrelated GPU-search score-normalization behavior outside this change.

Benchmarks

All measurements used cold FBIN sources, with zero source bytes resident before
the benchmark JVM.

External storage

WikiANN 10M, 768d:

Storage	Build time	Process writes	Recall at efSearch=1500
Mapped self-contained	112.039 s	32.852 GB	95.283%
External PREFETCH	85.058 s	2.132 GB	95.335%

External storage reduced build time by 24.08% and avoided 30.72 GB of writes.

Multi-segment builds

External PREFETCH, graph degree 32, intermediate degree 48, efSearch 1500:

Dataset	Segments	Build time	Mean search latency	Recall
Deep1B 10M, 96d	1	17.759 s	7.716 ms	97.495%
Deep1B 10M, 96d	4	14.019 s	16.623 ms	98.391%
Jasper 10M, 1536d	1	133.422 s	17.231 ms	98.310%
Jasper 10M, 1536d	4	73.379 s mean	37.922 ms mean	96.246% mean

Validation

• mvn -o spotless:check install: 424 tests, 0 failures/errors, 30 skipped
• Focused mapped/external multi-segment GPU suite: 7 tests, 0 failures/errors
• Benchmark harness: 42 tests, 0 failures/errors
• Spotless and git diff --check: passed

Coverage includes uneven slices, exact vector and global-ID preservation,
self-contained source deletion, external registration lifetime and relocation,
single-scan multi-segment validation, failure rollback, interruption,
cancellation, and concurrent metric aggregation.

Constraints

• This API builds Lucene101AcceleratedHNSWCodec indexes: GPU CAGRA construction followed by CPU HNSW search. It does not build CuVS2510GPUSearchCodec indexes.
• A one-shot build exclusively owns and recreates its normalized target directory. File-backed sources are resolved once and must remain immutable for the synchronous build.
• External indexes are not self-contained or readable by stock Lucene alone. Deployments must ship the Lucene directory and exact FBIN, register it through the same defining cuvs-lucene classloader, and retain at least one matching lease for every reader lifetime.
• PREFETCH performs read-ahead but trusts the supplied digest; VERIFY_SHA256 performs complete build-time verification.
• Multiple retained segments are supported, but later Lucene vector merges remain outside the bulk-index contract.
• checkIntegrity() currently hashes the shared external FBIN once per segment; normal reader opening and searching do not.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.