Skip to content

Reduce combinatorial GTest coverage using explicit size buckets and metric sharding #2578

Description

@divyegala

Reduce combinatorial GTest coverage using explicit size buckets and metric sharding

Problem

CUDA 13.3 CI on H100 spends most of its test time in a small number of parameterized ANN binaries:

CTest target H100 elapsed
NEIGHBORS_ANN_CAGRA_FLOAT_UINT32_TEST 1,680.87 s
NEIGHBORS_TEST 1,326.44 s
NEIGHBORS_ANN_CAGRA_HALF_UINT32_TEST 1,196.02 s
NEIGHBORS_ANN_IVF_PQ_TEST 1,085.10 s
NEIGHBORS_ANN_CAGRA_UINT8_UINT32_TEST 890.16 s
NEIGHBORS_ANN_CAGRA_INT8_UINT32_TEST 838.56 s
NEIGHBORS_ALL_NEIGHBORS_TEST 687.91 s before PR #2573
NEIGHBORS_ANN_IVF_FLAT_TEST 487.61 s
NEIGHBORS_ANN_VAMANA_TEST 422.94 s
NEIGHBORS_ANN_NN_DESCENT_TEST 287.47 s
NEIGHBORS_DYNAMIC_BATCHING_TEST 194.49 s

PR #2573 establishes the model for this work. It separates data-transfer testing from algorithmic coverage and replaces Cartesian products with low/medium/high diagonal cases. Its commit reports reducing NEIGHBORS_ALL_NEIGHBORS_TEST from approximately 1,200 to 228 generated cases and from approximately 34 minutes to 4 minutes.

Goal

Reduce PR CI time without removing:

  • metric-specific implementation coverage;
  • datatype coverage;
  • device/host/overlapped-transfer coverage;
  • explicit kernel-dispatch, alignment, batching, or capacity boundaries;
  • reproductions for previously reported bugs.

The central change is to stop testing every metric against every combination. Each metric must receive low-, medium-, and high-dimensional coverage, but the remaining axes should be distributed across metrics so each case has a unique purpose.

Global ANN size buckets

All ANN test families must use the same dimension vocabulary. Define the canonical values once in a shared test header (for example, cpp/tests/neighbors/ann_test_dimensions.hpp) and consume subsets from each ANN test generator. A family does not need to run every value; metrics select from the global catalog as described below.

Dimension buckets

Bucket Range Global catalog Purpose
D0: degenerate 1 1 Scalar/minimum-valid case; run once per ANN family, normally with L2
D1: low 2–31 aligned: 8, 16; unaligned: 7, 17 Small vector and first alignment transition
D2: medium 32–255 aligned: 64, 128; unaligned: 137 Common embedding sizes and first value above 128
D3: high 256–1023 aligned: 256, 512; unaligned: 619 Large vectors with aligned and unaligned coverage
D4: very high >=1024 aligned: 1024, 2048; unaligned: 1025, 2053 Very-large-vector path and padding coverage

The shared catalog is therefore:

{1, 7, 8, 16, 17, 64, 128, 137, 256, 512, 619, 1024, 1025, 2048, 2053}

Suggested shared constants:

inline constexpr std::array<uint32_t, 1> ann_dim_degenerate{1};
inline constexpr std::array<uint32_t, 4> ann_dim_low{7, 8, 16, 17};
inline constexpr std::array<uint32_t, 3> ann_dim_medium{64, 128, 137};
inline constexpr std::array<uint32_t, 3> ann_dim_high{256, 512, 619};
inline constexpr std::array<uint32_t, 4> ann_dim_very_high{1024, 1025, 2048, 2053};

Values outside the global catalog are removed by default. An ANN method may retain an extra value only as an explicitly documented regression/condition case under the exception policy below.

Dataset-row buckets

Bucket Range Default representative
R0: tiny 1–100 smallest valid value, plus documented regression/condition values only
R1: small 101–999 256 or nearest existing value
R2: medium 1,000–9,999 1,000 or 5,000
R3: large 10,000–99,999 10,000
R4: very large >=100,000 100,000; retain 131,072 only where it exercises a batching boundary

Query-count buckets

Bucket Range Default representative
Q0 1–16 10 or nearest existing value
Q1 17–256 100
Q2 257–4,096 1,000
Q3 4,097–99,999 10,000
Q4 >=100,000 retain only explicit grid/batch-splitting boundaries

Exception-only regression and condition cases

Global buckets provide all ordinary scale and alignment coverage. There are no algorithm-specific boundary overlays by default. A method-specific value outside the catalog is retained only when the test source explicitly identifies the algorithm bug or implementation condition it exercises.

An exception must have an adjacent source comment containing at least one of:

  • a linked issue/PR reproducer;
  • the exact dispatch or capacity condition being crossed, preferably with a source symbol or threshold;
  • the exact alignment/padding condition and why a global aligned/unaligned representative cannot exercise it.

If the comment only says “test various dimensions,” “large dimensions,” or “near a power of two,” the case does not qualify. Move it to the global buckets or remove it.

Family Existing exception candidates Required justification
IVF-PQ k 15/16, 31/32 when present, 63/65, 127/128, first value above 128, 256/257, 2048/2049 Top-k capacity, rounding, and fused/non-fused dispatch
IVF-PQ dimensions 512/513, 1023/1024/1025, 2048/2049/2050/2053 Power-of-two, padding, LUT, and kernel-dispatch coverage
IVF-Flat dimensions 2048–2056 currently listed Shared-memory limit and alignment coverage
IVF-Flat query counts 98,306, 100,000, 1,000,000 cases currently marked as batch/grid tests Grid-dimension and batch-splitting coverage
CAGRA small rows 31/32, only if separately documented Exact size-dependent branch or linked regression; proximity to 32 is insufficient
CAGRA dimensions 7/8, 128/137 Alignment and padding transitions
Vamana dimensions 512/619 High aligned and high unaligned paths
Filtered brute force k=2050/2051/2052; selectivity values that select SDDMM/gather/dense Select-k and filtered-search dispatch
Any suite A case with a bug/issue reference or a comment naming a dispatch path Regression coverage

Each qualifying exception is covered across CI shards with one metric and the minimum relevant datatype set. Do not cross it with all metrics, row counts, transports, build algorithms, or search algorithms.

A value is not protected merely because it is close to a power of two. The implementing PR must point to the dispatch/rounding branch, alignment class, or regression that makes it distinct. If all values select the same path, delete the extra values and rely on the global catalog.

Every applicable family receives high-dimensional unaligned coverage through global value 619 or 2053; it does not need a private unaligned value.

Apply any approved exceptions sparsely:

  • IVF-PQ dimension and k transition groups qualify only where their existing comments identify the rounding, capacity, or fused/non-fused condition. Distribute qualifying values across shards under L2.
  • IVF-Flat's “does not fit shared memory” comment justifies testing that condition, but not every value from 2048 through 2056. Use global {2048,2053} unless an extra residue has its own documented condition.
  • CAGRA 31/32 rows may remain only if the source comment identifies a size-dependent branch or bug. Otherwise use the global row bucket.
  • Filtered brute-force selectivity cases qualify because the source names the SDDMM, gather, and dense paths. Retain one case per proven path.

Deterministic metric sampling from the global buckets

For every family, each supported metric selects exactly one low, one medium, and one high/very-high value from the global catalog on each CI shard. Selection rotates across metrics and shards so aligned and unaligned representatives are covered globally.

Metric Samples per shard Additional responsibility
L2Expanded one from D1, one from D2, one from D3/D4 Canonical baseline; owns D0=1 and documented exception cases unless an exception is metric-specific
L2SqrtExpanded one from D1, one from D2, one from D3/D4 Square-root result path
InnerProduct one from D1, one from D2, one from D3/D4 Signed-score/reduced-precision path
CosineExpanded one from D1, one from D2, one from D3/D4 Normalization path and one host-input case
L1 one from D1, one from D2, one from D3/D4 L1-specific distance path
BitwiseHamming one aligned value from each applicable bucket Bit-packed/integer path

If an ANN method does not support a selected dimension, select the nearest supported value in the same bucket and alignment class. If none exists, document the missing bucket rather than pulling an arbitrary dimension from another bucket.

Shard-selection algorithm

Independent CI jobs do not communicate. Coordination comes entirely from three identical inputs supplied by the CI matrix: the head commit SHA, the total number of coverage slots, and a unique explicit slot number. Job start order, completion order, runner hostname, and physical GPU identity must not affect selection.

Do not call a nondeterministic RNG or use std::hash during GTest registration. std::hash is not a cross-build persistence contract. Implement a small specified stable hash such as 64-bit FNV-1a over UTF-8 bytes, with NUL separators between fields. Include a schema version so a deliberate future mapping change is visible:

key    = "cuvs-ann-matrix-v1\0" + head_commit_sha + "\0" +
         ann_family + "\0" + metric + "\0" + bucket_or_axis
h      = fnv1a64(key)
pool   = the sorted supported values in that global bucket or documented exception set
offset = h % pool.size()
index  = (offset + coverage_slot) % pool.size()
value  = pool[index]

This is random-looking rotation, not independent random sampling. If slots 0..7 all run, every pool of size at most eight is covered with certainty: a four-value pool is selected exactly twice, a three-value pool is selected two or three times, and a seven-value high/very-high pool is selected at least once. Different families, metrics, buckets, and axes receive different offsets because they use different hash keys.

Required environment/configuration:

CUVS_TEST_MATRIX_MODE  = sharded
CUVS_TEST_SHARD_INDEX  = 0..CUVS_TEST_SHARD_COUNT-1
CUVS_TEST_SHARD_COUNT  = 8
CUVS_TEST_MATRIX_SEED  = pull-request head SHA, or ordinary commit SHA outside a PR

Define the coverage slot as an explicit CI matrix value. Never derive it from CUDA_VISIBLE_DEVICES, the runner hostname, a job counter, matrix execution order, or the physical GPU ordinal; independent jobs commonly all see their assigned GPU as device zero, and scheduling order is unstable. A GitHub Actions-shaped example is:

strategy:
  matrix:
    coverage_slot: [0, 1, 2, 3, 4, 5, 6, 7]
env:
  CUVS_TEST_MATRIX_MODE: sharded
  CUVS_TEST_SHARD_INDEX: ${{ matrix.coverage_slot }}
  CUVS_TEST_SHARD_COUNT: 8
  CUVS_TEST_MATRIX_SEED: ${{ github.event.pull_request.head.sha || github.sha }}

If the eight jobs already exist, assign one slot to each existing job; do not create eight additional replicas solely for this mechanism. Slot eligibility is target-specific: a target receives the eight-slot guarantee only if that target is built and executed in all eight slots. For a conditionally built target such as HNSW, either define eight eligible HNSW slots, use the actual smaller eligible count with pools no larger than that count, or run that target in full mode. The manifest validator checks coverage separately for each target, not merely across the workflow as a whole.

Use the PR head SHA rather than a platform-generated merge SHA when the goal is repeatability for the same submitted revision. A retry of a job receives the same slot and seed and must generate the identical GTest list. A new commit intentionally changes the selection.

Local and unconfigured execution must fail safe:

  • CUVS_TEST_MATRIX_MODE=full enumerates the complete catalog for manual/nightly validation.
  • If the mode is unset, default to full, not slot zero.
  • sharded mode must fail immediately if the seed, count, or index is absent, if the count is not eight, or if the index is outside [0, count).

Each binary must print the schema version, seed, slot/count, and selected parameter names. Each CI job uploads its --gtest_list_tests output as a small manifest. A final non-GPU validation job checks that slots 0..7 are present exactly once and that their union satisfies the bucket, metric, datatype, enum, and documented-exception invariants. This validation detects a missing or accidentally duplicated CI slot; it is not required for the jobs to choose their cases.

Rerunning the same commit and slot must select the same cases. Across eight slots, every bucket containing at most eight values is covered at least once. If these CTest binaries execute only once across an eight-GPU resource pool rather than once in each of eight replicated jobs, this scheme must not be enabled until the CI topology provides eight explicit coverage slots for the target.

Independent random selection is not sufficient: with a four-value bucket and eight independent draws, the probability of covering all four values is only about 62%. The shard permutation provides 100% coverage for that bucket when all eight shards run.

Do not independently hash every factor in a case when pairwise coverage between factors is required. First define the sparse row identities (for example, CAGRA's 15 metric x search_algorithm rows), then use the slot/hash rotation only to assign dimensions, row scales, build modes, and boolean values to those rows. The eight-shard union must be checked against the explicit pairwise invariants below.

For the three cases assigned to a metric, rotate the other axes:

Scale Row scale Input location Layout/other axis
Low dimension R2/medium rows device default layout
Medium dimension R3/large rows host alternate layout/build path
High dimension R4/very-large rows overlap/prefetch when supported remaining build/search path

Shift this mapping by one position for the next metric. The result must satisfy:

  1. On each shard, every metric appears in low, medium, and high/very-high dimension scales using the global catalog.
  2. Across all shards, every supported value in every global bucket and documented exception set appears at least once when the pool size is no larger than the shard count.
  3. Every row scale and input-location mode appears at least once globally.
  4. Every supported build/search/layout enum appears at least once globally.
  5. L2 owns documented exception cases unless the underlying condition is metric-specific.
  6. Non-L2 metrics do not receive the full boundary or transport matrix.

Do not create metric × dimension × rows × transport × build × search Cartesian products. The shared catalog defines available values; it is not a requirement that every metric or every ANN family run all catalog values.

Phase 1: deterministic low-risk edits

These changes do not require designing a covering array.

1. CAGRA

File: cpp/tests/neighbors/ann_cagra.cuh

  • In the dimension-specific product block, change:

    {1, 3, 5, 7, 8, 17, 64, 128, 137, 192, 256, 512, 1024}
    

    to:

    {1, 7, 8, 17, 64, 128, 137, 256, 512, 1024}
    

    Removed same-bucket representatives: 3, 5, 192.

  • Do not remove dim=619 from the separate non-owning-memory-buffer block. That case supplies CAGRA's high-dimensional unaligned coverage.

  • In the small-dataset row-count block, change:

    {3, 6, 31, 32, 64, 101}
    

    to:

    {3, 101}
    

    Removed same-regime row counts: 6, 31, 32, and 64. The source labels these only as small-dataset corner cases; it does not identify a 31/32 algorithm boundary, dispatch condition, or linked regression, so they do not qualify as exceptions.

  • Expected static reduction: approximately 2,184 generated GTests across the float, half, int8, and uint8 CAGRA binaries: approximately 936 from the dimension block and 1,248 from the row-count block.

2. Vamana

File: cpp/tests/neighbors/ann_vamana.cuh

In all four product blocks, change:

{1, 3, 5, 7, 8, 17, 64, 128, 137, 192, 256, 384, 512, 619, 1024}

to:

{1, 7, 8, 17, 64, 128, 137, 256, 512, 619, 1024}

Removed same-bucket representatives: 3, 5, 192, 384. Retain 619 as the only unaligned Vamana dimension above 137.

Expected static reduction: 52 inputs per datatype, 208 generated GTests total.

3. IVF-PQ exact/default-equivalent duplicates

Files:

  • cpp/tests/neighbors/ann_ivf_pq.cuh
  • cpp/tests/neighbors/ann_ivf_pq/test_float_int64_t.cu
  • cpp/tests/neighbors/ann_ivf_pq/test_int8_t_int64_t.cu
  • cpp/tests/neighbors/ann_ivf_pq/test_uint8_t_int64_t.cu

enum_variety() contains five rows that re-specify defaults:

codebook_kind = PER_SUBSPACE
pq_bits = 8
force_random_rotation = false
lut_dtype = CUDA_R_32F
internal_distance_dtype = CUDA_R_32F

Retain one baseline row per metric and remove the other four equivalent rows. Where defaults() already supplies the L2 baseline, remove all five equivalent enum rows for L2.

For uint8, remove enum_variety_l2() entirely because the raw enum_variety() already uses the default L2Expanded metric and produces the same 17 inputs.

Expected static reduction: approximately 233 generated GTests.

Important: normalize the retained baseline's min_recall explicitly. The removed rows differ from the default primarily in their acceptance threshold, not the code-under-test configuration.

4. IVF-Flat

File: cpp/tests/neighbors/ann_ivf_flat.cuh

  • Remove one of the two identical small-dimension rows with dim=5, metric=CosineExpanded, and adaptive_centers=false.

  • Leave the dim=2048–2056 boundary block unchanged.

  • Leave the explicitly documented batch/grid and radix-sort cases unchanged.

  • Reduce each 14-row host-input and host-prefetch block to these eight workload/metric pairs:

    {1000,  10000,  16, 10, 40, 1024, L2Expanded}
    {1000,  10000,  16, 10, 40, 1024, CosineExpanded}
    {100,   10000,  16, 10, 20,  512, L2Expanded}
    {100,   10000,  16, 10, 20,  512, CosineExpanded}
    {1000, 100000,  16, 10, 20, 1024, L2Expanded}
    {1000, 100000,  16, 10, 20, 1024, CosineExpanded}
    {10000,131072,   8, 10, 20, 1024, L2Expanded}
    {10000,131072,   8, 10, 20, 1024, CosineExpanded}
    

    Preserve the appropriate host_dataset=true and prefetch/overlap flag for each block.

    Remove from each transport block:

    • n_probes=50 and 70 repetitions at the otherwise identical 1000 × 10000 × dim=16 workload;
    • the 20-query × 100000-row L2/cosine pair, since the retained 1000-query × 100000-row pair covers the same row bucket and path.

Expected static reduction: approximately 52 generated GTests: 48 transport-matrix cases plus four datatype replications of the exact duplicate.

5. Dynamic batching

File: cpp/tests/neighbors/dynamic_batching.cuh

Change the one-factor sweeps to:

n_queries              = {10, 100}             // remove 50
k                      = {100}                 // remove 200
max_batch_size         = {4,16,128,512,1024}  // remove 256
n_queues               = {1,16}               // remove 2,32
max_concurrent_threads = {1,16}               // remove 2,32

Retain both values of conservative_dispatch for every remaining input.

Expected static reduction: 14 cases per backend, 56 generated GTests across four backends.

6. NEIGHBORS_TEST

File: cpp/tests/neighbors/brute_force.cu

Remove the second identical row-major L2SqrtExpanded entry:

{256, 512, 16, 8, L2SqrtExpanded, true, false}

Expected static reduction: 2 generated GTests because the vector is instantiated for float and half.

Do not reduce brute_force_prefiltered.cu in Phase 1. Its close k=2050/2051/2052 values and filter selectivities deliberately select different dispatch paths.

Phase 2: remove metric Cartesian products

Apply the metric-sharding policy to every reducible family in the complete estimate below: CAGRA, all-neighbors, IVF-PQ, NN-descent, Vamana, IVF-Flat, HNSW, the parameterized components of NEIGHBORS_TEST, IVF-SQ, HNSW-ACE, ScaNN, dynamic batching, IVF-RaBitQ, tiered index, and ANN brute force. Leave the explicitly listed regression/condition and already-sparse matrices unchanged.

CAGRA acceptance boundary

The first CAGRA product currently contains 720 base inputs. Replace it with exactly 15 inputs per shard. Fifteen is the minimum because five metrics must each appear once with each of three search algorithms. Assign the remaining axes to those 15 rows as follows:

  • on every shard, every one of the five metrics appears with both dim=1 and dim=16 at least once;
  • on every shard, every metric appears exactly once with each search algorithm;
  • on every shard, every metric appears with each build algorithm at least once;
  • degrees 32, 47, and 64 each appear under both build algorithms globally;
  • query-size settings 0 and 10 each appear under every search algorithm globally;
  • both values of the boolean flag currently swept at the end of the product appear with every metric on every shard;
  • L2 owns any remaining full transition/boundary combinations;
  • degree, build, query-size, dimension, and boolean assignments rotate deterministically by shard so all required pairs appear in the eight-shard union.

Target: reduce this block from 720 to 15 base inputs per shard (97.9% fewer) without removing a value from any axis across the eight-shard union.

Other metric matrices

For each remaining matrix:

  • retain three core cases per metric: one D1/low case, one D2/medium case, and one D3-or-D4/high case; retain D0=1 once per ANN family under L2;
  • add documented regression/condition exceptions under L2 only, unless the condition is metric-specific;
  • add one metric-specific case for cosine normalization, inner-product signed scoring, L2-sqrt transformation, L1, and Hamming where supported;
  • test host and overlap paths in a separate matrix using L2 and cosine only;
  • do not exceed 3 × number_of_supported_metrics + documented_exception_cases + transport_cases.
  • sample documented exception sets by shard as specified above; do not instantiate every exception value on every shard.

Complete static estimate

Scope and interpretation

This estimate covers every value-parameterized GTest in every NEIGHBORS_* CTest target represented in the supplied CUDA 13.3 run, at repository commit e8d409b3ac2a61488f807732c84e0462b59c6fe9. It counts the expansion of parameter objects through their TEST_P bodies. Non-parameterized tests are not counted, and cluster, distance, preprocessing, stats, and utility targets are unchanged because the global ANN bucket policy does not apply to them.

“After” means cases generated on one CI shard. Different shards contain the same number of cases but select different global representatives. The eight-shard union therefore has broader value coverage without making an individual GPU run the full matrix.

Family / CTest target, sorted by cases removed Current Proposed per shard Removed Reduction
Four main CAGRA datatype targets 17,927 1,049 16,878 94.1%
NEIGHBORS_ALL_NEIGHBORS_TEST 1,560 228 1,332 85.4%
NEIGHBORS_ANN_IVF_PQ_TEST 1,397 414 983 70.4%
NEIGHBORS_ANN_NN_DESCENT_TEST 880 61 819 93.1%
NEIGHBORS_ANN_VAMANA_TEST 780 52 728 93.3%
NEIGHBORS_ANN_IVF_FLAT_TEST 388 92 296 76.3%
NEIGHBORS_HNSW_TEST 256 24 232 90.6%
NEIGHBORS_TEST parameterized components 337 166 171 50.7%
NEIGHBORS_ANN_IVF_SQ_TEST 134 30 104 77.6%
Four HNSW-ACE datatype targets 144 40 104 72.2%
NEIGHBORS_ANN_SCANN_TEST 105 30 75 71.4%
NEIGHBORS_DYNAMIC_BATCHING_TEST 160 104 56 35.0%
NEIGHBORS_ANN_IVF_RABITQ_TEST 138 84 54 39.1%
NEIGHBORS_TIERED_INDEX_TEST 72 21 51 70.8%
NEIGHBORS_ANN_BRUTE_FORCE_TEST 78 48 30 38.5%
Explicit regression/condition and already-sparse parameterized targets 188 188 0 0%
Complete parameterized-neighbors total 24,544 2,631 21,913 89.3%

If all eight GPU jobs currently execute the complete matrices, aggregate executions change from 24,544 * 8 = 196,352 to 2,631 * 8 = 21,048, removing 175,304 executions while the eight jobs rotate through the catalog.

Reproducible proposed-count budgets

The estimates above use the following concrete budgets. An implementing agent may produce fewer cases only if all stated coverage invariants still hold; it must not exceed these budgets by recreating a Cartesian product.

Family Proposed-count derivation
CAGRA Main generate_inputs(): 62 base inputs = 15 leading categorical cases + 5 logical-merge + 4 additional-distance + 6 small-dataset + 13 metric/dimension/build + 6 team/build + 3 team-size + 2 row/host + 6 refinement + 2 non-owning-buffer. Current 13 fixture/test-body replications produce 62 * 13 = 806. Add-node becomes 6 * 4 = 24; filtering becomes 9 * 3 = 27; the already-documented multi-partition matrix stays 24 * 8 = 192. Total: 806 + 24 + 27 + 192 = 1,049.
All-neighbors Use PR #2573's separated sparse matrices unchanged: 228 generated cases. Against the current source expansion of 1,560, this removes 1,332.
IVF-PQ Float: (13 metric/size core + 17 enum configurations distributed across metrics) * 6 test bodies + 4 flat-layout cases = 184. Int8: (10 metric/size core + 17 distributed enum configurations + 7 documented k regimes) * 5 = 170. Uint8: (13 metric/size core + 17 distributed enum configurations) * 2 = 60. Total: 414. No enum configuration is discarded; it is assigned to one metric instead of all metrics.
NN-descent Shared main vector: 3 * (3 * 6 metrics + D0) = 57 across three datatypes. Distance-epilogue matrix: four pairwise cases covering both metrics, row counts, dimensions, and degrees. Total: 61.
Vamana Preserve its 13 distinct graph-degree/visited-size/queue combinations, but assign one global-bucket dimension to each combination instead of crossing all 15 dimensions. The same 13-case diagonal runs for four datatypes: 13 * 4 = 52.
IVF-Flat Per datatype: 23 = 13 metric/size core + 4 dedicated host/overlap cases + 3 documented grid/batch-splitting cases + 1 radix-sort case + the 2-case recall comparison. The shared-memory condition is covered by global high values 2048 and 2053 inside the core. 23 * 4 datatypes = 92.
HNSW Per datatype: 2 metrics * 3 dimension scales = 6, rotating the two row counts and two graph degrees. 6 * 4 datatypes = 24.
NEIGHBORS_TEST Fixed example 2 + random brute force 37 * 2 datatypes = 74 + prefiltered brute force 14 * 4 filter/type fixtures = 56 + sparse case 1 + refine 6 * 3 datatypes = 18 + distance-NN 15 = 166. The 14 prefiltered inputs comprise 12 metric/scale cases plus k=0 and k=1; the documented SDDMM/gather/dense paths must be embedded in those 12 cases.
IVF-SQ Float: 26 = 13 metric/size core + 3 k regimes + 3 nprobe/nlist relationships + 3 row/query scales + 2 recall-stability cases + 2 host-transfer cases. Half: one case per supported metric = 4. Total: 30. The documented k=257 materialized-fallback case is mandatory.
HNSW-ACE Per datatype: six metric/scale cases plus the four existing explicit invalid-partition, memory-fallback, and spill inputs. 10 * 4 = 40. This deliberately leaves the exception set unmodified. A later change may move dtype-invariant exception cases to float only, but that saving is not included here.
ScaNN Six dimension cases (2 PQ widths * 3 scales) plus four distinct BF16/AVQ/SOAR feature cases = 10 parameters. The three existing test bodies produce 30 GTests.
Dynamic batching Keep the already-specified 26-case matrix per backend, including both conservative-dispatch modes. 26 * 4 = 104.
IVF-RaBitQ Per test body: 28 = 4 dimension scales + 3 nprobe scales + 4 k scales + all 9 compression-bit values + all 8 search-mode/one-bit cases. 28 * 3 = 84.
Tiered index Per backend: 2 metrics * 3 scales + D0 = 7, rotating both row counts and both extend/merge strategies. 7 * 3 backends = 21.
ANN brute force Per datatype: 24 = 13 metric/size core + 6 fused-L2 cases (three scales for expanded and unexpanded) + 5 tile/backend cases. 24 * 2 = 48.
Unchanged CAGRA bug reproducers, CAGRA UDF-filter, batch-load-iterator and merge-condition parameters, ball-cover, epsilon-neighborhood, and multi-GPU ANN already describe distinct conditions or are already sparse. Their 188 parameterized GTests remain.

Runtime interpretation

The parameterized neighbors targets account for approximately 9,488 seconds (158.1 minutes) in the supplied serial H100 run. Scaling every target linearly by its generated-case ratio gives a mechanical planning estimate of approximately 1,787 seconds (29.8 minutes), 81.2% less. This is not an acceptance target: CAGRA's removed low-dimensional cases may be cheaper than its retained high-dimensional cases, while removing the largest IVF and prefiltered-brute-force workloads may save disproportionately more. Only before/after H100 measurements can establish the actual runtime reduction.

For a conventional full matrix containing M metrics, three dimension scales, three row scales, and three transport modes, the old shape is 27M cases. The new core is 3M + 1 cases, before documented exceptions. This is an 88.1% core-matrix reduction for five metrics (135 -> 16) and 88.3% for six (162 -> 19). Exceptions add only their minimal reproducer and are never multiplied by the other axes.

Verification

The implementing PR must include:

  • Before/after output from --gtest_list_tests for every modified CTest binary, including the per-shard count and eight-shard union.
  • Before/after elapsed time on one H100 using CUDA 13.3.
  • A table mapping every removed input to its retained bucket representative.
  • Evidence that all documented dispatch boundaries remain in the eight-shard union.
  • Evidence that every supported metric has low-, medium-, and high-dimension coverage.
  • Every applicable family retains at least one unaligned dimension above 137.
  • Ordinary ANN dimension sweeps use the shared global catalog; every value outside it is a documented exception tied to a named condition or regression.
  • Evidence that every datatype and supported build/search/transport enum remains covered.
  • All modified CTest targets pass three consecutive runs.
  • Rerunning the same commit with the same shard index produces the identical GTest parameter list.
  • No recall threshold is weakened solely to make the reduced matrix pass.

Success criteria

  • Phase 1 lands independently and removes the deterministic duplicate/same-bucket cases listed above.
  • Phase 2 reduces the first CAGRA 720-case product to exactly 15 inputs per shard.
  • The four CAGRA binaries, IVF-PQ, IVF-Flat, Vamana, dynamic batching, and NEIGHBORS_TEST show a meaningful aggregate H100 runtime reduction.
  • NEIGHBORS_ALL_NEIGHBORS_TEST remains consistent with the sparse-diagonal structure introduced by PR Reduce NEIGHBORS_ALL_NEIGHBORS_TEST combinatorial test space #2573.
  • No documented regression/condition case, metric-specific path, datatype, or transport mode is lost.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions