Forward-merge release/26.10 into main - #2589
Open
rapids-bot[bot] wants to merge 7 commits into
Open
Conversation
…#2564) Follow-up to #2548 Contributes to rapidsai/build-planning#324 * switches to wheel builds and `pip` devcontainers getting `tileiras` from the system-installed CTK, not `cuda-toolkit` wheels. ## Notes for Reviewers ### Benefits of the `tileiras` change * one less version to need to remember to update when we bump the CTK version we build against * consistent with how we get NVCC from the system install, not wheels * removes a source of patching in DLFW builds (where a system CTK is always preferred to one provided by wheels) ### How I tested this relied on CI
Contributor
Author
|
FAILURE - Unable to forward-merge due to an error, manual merge is necessary. Do not use the IMPORTANT: When merging this PR, do not use the auto-merger (i.e. the |
- remove stream pool creation from individual wrappers Authors: - Tarang Jain (https://github.com/tarang-jain) - Mike Sarahan (https://github.com/msarahan) Approvers: - Artem M. Chirkin (https://github.com/achirkin) - Bradley Dice (https://github.com/bdice) URL: #2526
`testMergeTwoSegsWithASingleDocPerSeg` and `testTwoVectorFieldsPerDoc` asserted that ordinal i of the merged segment holds the i-th document's vector. Lucene does not offer that: `MockRandomMergePolicy` shuffles the segments of a forced merge on purpose, so the document committed second can land at ordinal 0. Stock `Lucene99HnswVectorsFormat` fails the same tests on the same seeds, so no cuVS writer is involved. The assertions now resolve each ordinal to its document and check that the document kept its own vector, which leaves the randomized merge policy in play. ordToDoc is checked to be increasing so the id lookup cannot agree with a mapping that is itself garbled. Applied to the quantized and GPU-search formats too, which carried the same assumption unreported. Closes #2550 Authors: - Igor Motov (https://github.com/imotov) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #2556
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2589 +/- ##
=======================================
Coverage 86.93% 86.93%
=======================================
Files 33 33
Lines 176 176
=======================================
Hits 153 153
Misses 23 23 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…)" (#2597) This reverts commit 6a5bba0. After some additional discussion, we concluded that while this change addresses a very specific use case well, there's an opportunity to make it more generic and more widely applicable. We'll be working on a solution that covers a broader set of use cases and open a follow-up PR soon. Authors: - Igor Motov (https://github.com/imotov) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #2597
Replaces #2443 Containerizes the process of building the standalone C library tarballs The starting point for this PR is @cjnolet 's work in #2443, where he wrote this description: > _This PR is based on @msarahan's original POC, with the logic of the ci sript moved into build.sh and invoked through the CI script. The readme is also moved into the build and install guide in the docs._ Changes relative to that PR: * threads `PARALLEL_LEVEL` throught so builds are parallelized * enables the build cluster (`sccache-dist`) in CI * simplifies configuration flow (e.g. reduced duplication of default values, removal of unnecessary flexibility) * makes flow of AWS creds for `sccache` more secure * removes unnecessary configuration * removes unnecessary package installs, consolidates package installs * removes an unnecessary `git clone` of https://github.com/rapidsai/spdx-license-builder * enables `sccache` in CI and with a pattern that could work locally (will share details privately) * updates GitHub Actions third-party actions to their latest versions ## Notes for Reviewers ### How I tested this Locally tried each of the new commands added to `build.md`, with `sccache` enabled. Tried just `build.sh` without `sccache` enabled (that takes a lot longer to run). <details><summary>code for flipping between those (click me)</summary> ```shell # enabling sccache export CI=true # (private steps setting up 'sccache' profile) AWS_ACCESS_KEY_ID=$( aws configure get aws_access_key_id \ --profile sccache ) AWS_SECRET_ACCESS_KEY=$( aws configure get aws_secret_access_key \ --profile sccache ) AWS_SESSION_TOKEN=$( aws configure get aws_session_token \ --profile sccache ) export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN # testing without 'sccache' unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN CI ``` </details> ```shell clean() { sudo rm -rf ./{build,c/build/,cpp/build,libcuvs_c.tar.gz} } # all defaults clean ./build.sh tarball # customizing base image clean CUVS_TARBALL_CUDA_VERSION=12.9.2 \ CUVS_TARBALL_PYTHON_VERSION=3.11 \ ./build.sh tarball # customizing output directory, building tests clean CUVS_TARBALL_BUILD_OUTPUT_DIR="${PWD}/dist" ./build.sh tarball --tarball-build-tests # manual run (no build.sh) clean docker build \ -f Dockerfile.standalone \ --build-arg CUDA_VERSION="13.3.0" \ --build-arg PYTHON_VERSION="3.14" \ --build-arg RAPIDS_VERSION="$(head -1 ./VERSION | cut -d. -f1,2 )" \ -t cuvs-standalone-c:local \ . mkdir -p "${PWD}/dist" docker run --rm \ -v "${PWD}:/workspace" \ -v "${PWD}/dist:/build" \ cuvs-standalone-c:local --tarball-build-tests ``` Saw high cache hit rates from `sccache` and everything working as expected. Also looked at CI logs and saw that fully-cached jobs take around 30 minutes, pretty similar to the timings in CI today. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Mike Sarahan (https://github.com/msarahan) - Corey J. Nolet (https://github.com/cjnolet) URL: #2595
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| fetch-depth: 0 | ||
| persist-credentials: false | ||
| - name: Standardize repository information | ||
| uses: rapidsai/shared-actions/rapids-github-info@main |
| aws-region: ${{ vars.AWS_REGION }} | ||
| role-duration-seconds: 10800 | ||
| - name: Setup sccache-dist | ||
| uses: rapidsai/shared-actions/setup-sccache-dist@main |
| aws-region: ${{ vars.AWS_REGION }} | ||
| role-duration-seconds: 10800 | ||
| - name: Setup sccache-dist | ||
| uses: rapidsai/shared-actions/setup-sccache-dist@main |
The C++ core already builds and is tested on both amd64 and arm64; cuvs-java had a hardcoded amd64-only gate that cuvs-lucene inherited. Java bytecode is portable, but cuvs-java isn't pure Java — it uses the Panama FFM API to call into native libcuvs_c.so, and those bindings are generated by jextract, which bakes struct/function ABI layouts (computed by parsing the C headers with clang for the host's target triple) into the generated .class files at build time. Reusing an amd64-generated binding on aarch64 should work, since no arch-conditional logic in the jextract-visible header surface changes any type layout, struct offset, or function signature between x86_64 and aarch64 Linux. We therefore expect the plain (no-native-bundled) jar to be arch-portable in practice. To verify that, the aarch64 cuvs-lucene CI job installs the amd64-built jar and runs cuvs-lucene's full test suite against it on aarch64 hardware with an aarch64-native libcuvs_c.so. Closes #1236. Authors: - Igor Motov (https://github.com/imotov) Approvers: - James Lamb (https://github.com/jameslamb) - MithunR (https://github.com/mythrocks) URL: #2541
| needs: [conda-cpp-build, changed-files] | ||
| permissions: | ||
| contents: read | ||
| uses: rapidsai/shared-workflows/.github/workflows/compute-matrix.yaml@release/26.10 |
| packages: read | ||
| pull-requests: read | ||
| secrets: inherit # zizmor: ignore[secrets-inherit] | ||
| uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@release/26.10 |
| packages: read | ||
| pull-requests: read | ||
| secrets: inherit # zizmor: ignore[secrets-inherit] | ||
| uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@release/26.10 |
| # architecture is automatically picked up here without a workflow change. | ||
| permissions: | ||
| contents: read | ||
| uses: rapidsai/shared-workflows/.github/workflows/compute-matrix.yaml@release/26.10 |
| packages: read | ||
| pull-requests: read | ||
| secrets: inherit # zizmor: ignore[secrets-inherit] | ||
| uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@release/26.10 |
| packages: read | ||
| pull-requests: read | ||
| secrets: inherit # zizmor: ignore[secrets-inherit] | ||
| uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@release/26.10 |
…2521) ## Summary Track the coordinated migration of stream APIs and call sites from `rmm::cuda_stream_view` to CCCL's `cuda::stream_ref`. This propagates `cuda::stream_ref` through RMM containers and memory resources, RAFT resource and handle APIs, downstream C++ interfaces, Python/Cython bindings, benchmarks, tests, and documentation. This migrates affected cuVS API signatures and internal call sites while extracting raw stream handles only where CUDA, generated/JIT, or legacy APIs require them. Depends on rapidsai/rmm#2372 and NVIDIA/raft#3129. Tracked in rapidsai/build-planning#318. ## Migrations - Pass `cuda::stream_ref` through stream pools, resource accessors, conditionals, and downstream APIs without converting to `rmm::cuda_stream_view` - Use `cuda::stream_ref` constructions for default/legacy/per-thread streams - `rmm::cuda_stream_default` ➡️ `cuda::stream_ref{cudaStream_t{cudaStreamDefault}}` - `rmm::cuda_stream_legacy` ➡️ `cuda::stream_ref{cudaStreamLegacy}` - `rmm::cuda_stream_per_thread` ➡️ `cuda::stream_ref{cudaStreamPerThread}` - Use `.get()` when calling an API that requires a raw `cudaStream_t`, including CUDA runtime, library, CUB, and legacy API boundaries (previously `rmm::cuda_stream_view` used `value()`) - Use `.sync()` when synchronizing a `cuda::stream_ref` (previously `rmm::cuda_stream_view` used `synchronize()`) - Update Cython declarations and call sites to pass stream references directly where supported Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Divye Gala (https://github.com/divyegala) URL: #2521
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-merge triggered by push to release/26.10 that creates a PR to keep main up-to-date. If this PR is unable to be immediately merged due to conflicts, it will remain open for the team to manually merge. See forward-merger docs for more info.