feat(hipcub): use primbench

[Nemi26]

Motivation

hipCUB's benchmarks have been updated to use primbench. primbench performs repeated GPU-specific measurement, cache control, warmup handling, and noise analysis that are difficult to express using Google Benchmark.

Because primbench measures cold-cache execution by default, benchmark throughput numbers are generally expected to be lower than previous Google Benchmark results due to cache clearing between measurement batches.

The diff of benchmark_device_batch_copy.cpp represents what changes have been made to all benchmarks.

JIRA ID : ROCM-27706

Technical Details

Caution

This PR must not be merged before the PR Add hipCUB CCCL 3.0.x support on which it is based has been merged.

After that CCCL PR has been merged, this primbench PR should be rebased onto the develop branch, starting with its first true commit use primbench in all the benchmarks.

Important

AMD should confirm that the HIP benchmarks build and run successfully on Windows, since Stream HPC's Windows CI is currently broken.

Note

If AMD's GPU cooling setup is too weak or too powerful then --min-gpu-temp (default 50°C) or --max-gpu-temp (default 60°C) will need to be passed to benchmarks by AMD's CI. I recommend holding off on passing these flags until a benchmark actually times out from not reaching the 50°C-60°C temperature range.

The easiest way to review this PR is commit-by-commit starting from commit feat(hipcub): use primbench in benchmark_device_batch_copy.cpp. The commit list below serves as an overview of the changes. To walk through their code I recommend just repeatedly clicking the Next button that is shown when viewing a specific commit:

feat(hipcub): use primbench in benchmark_device_batch_copy.cpp
feat(hipcub): Google Benchmark is no longer a dependency
feat(hipcub): use primbench in 6 block algorithms
feat(hipcub): use primbench in 5 block algorithms
feat(hipcub): use primbench in 5 device algorithms
feat(hipcub): use primbench in 5 device algorithms
feat(hipcub): use primbench in 7 device algorithms
feat(hipcub): use primbench in 6 warp algorithms

Test Plan

The below blocks of commands can be copied directly into a terminal, so you don't need to copy the lines individually.

See the Test Result header in this description for the zip that contains all of the scripts that these reproduction steps use.

VS Code dev containers

HIP Dockerfile + devcontainer.json

Dockerfile:

FROM rocm/rocm-terminal:latest

devcontainer.json:

{
    "build": {
        "dockerfile": "Dockerfile"
    },
    "name": "hip-minimal",
    "runArgs": [
        "--device=/dev/kfd",
        "--device=/dev/dri"
    ]
}

CUDA Dockerfile + devcontainer.json

Dockerfile:

FROM nvidia/cuda:12.9.1-devel-ubuntu24.04

RUN apt update && apt install -y git cmake ninja-build wget

RUN wget -qO- https://repo.radeon.com/rocm/rocm.gpg.key | gpg --dearmor > /etc/apt/keyrings/rocm.gpg

RUN echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/7.2 noble main" > /etc/apt/sources.list.d/rocm.list

RUN tee /etc/apt/preferences.d/rocm-pin-600 <<'EOF'
Package: *
Pin: origin repo.radeon.com
Pin-Priority: 600
EOF

RUN apt update && apt install -y hip-base

ENV HIP_PLATFORM=nvidia

devcontainer.json:

{
    "build": {
        "dockerfile": "Dockerfile"
    },
    "name": "cuda-minimal",
    "privileged": true,
    "runArgs": [
        "--gpus=all"
    ]
}

Reproduction steps

Installing dependencies

sudo apt update && \
sudo apt install -y cmake ninja-build

Cloning

git clone --no-checkout --filter=blob:none https://github.com/ROCm/rocm-libraries.git && \
cd rocm-libraries && \
git sparse-checkout init --cone && \
git sparse-checkout set projects/hipcub shared/primbench

Checking out this pull request

git fetch origin pull/8534/head:users/Nemi26/use-primbench-in-hipcub && \
git checkout users/Nemi26/use-primbench-in-hipcub

Building primbench benchmarks

Setup

cd projects/hipcub && \
mkdir build_primbench && \
cd build_primbench

HIP

export gfx=$(rocm_agent_enumerator | head -n1) && \
CXX=hipcc cmake -GNinja -DBUILD_BENCHMARK=ON -DGPU_TARGETS="$gfx" .. && \
ninja && \
cd ..

CUDA

export gfx=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader,nounits | head -n 1 | tr -d '.') && \
cmake -GNinja -DBUILD_BENCHMARK=ON .. && \
ninja && \
cd ..

Copying run_benchmarks_new.py to a server

scp \
  ~/Downloads/run_benchmarks_new.py \
  myname@myserver:~/rocm-libraries/projects/hipcub

Running primbench benchmarks

mkdir results_primbench && \
python3 run_benchmarks_new.py \
  --benchmark-executables-dir=build_primbench/benchmark \
  --gpu-architecture=$gfx \
  --json-out-dir=results_primbench \
  --csv-out-dir=results_primbench

Copying primbench benchmark results from a server

scp \
  -r \
  myname@myserver:~/rocm-libraries/projects/hipcub/results_primbench \
  ~/Downloads

Checking out the baseline develop commit which this PR is based on

If PR Add hipCUB CCCL 3.0.x support has not been merged yet, you will first need to check it out:

git fetch origin pull/4079/head:users/matyas-streamhpc/hipcub-cccl-3-0

You can then check its last commit out:

# You can change this commit hash to `develop`, once the CCCL 3.0.x PR is merged
git checkout e7f31172c14

Building develop benchmarks

HIP

cd projects/hipcub && \
mkdir build_develop && \
cd build_develop && \
export gfx=$(rocm_agent_enumerator | head -n1) && \
CXX=hipcc cmake -GNinja -DBUILD_BENCHMARK=ON -DGPU_TARGETS="$gfx" .. && \
ninja && \
cd ..

CUDA

cd projects/hipcub && \
mkdir build_develop && \
cd build_develop && \
export gfx=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader,nounits | head -n 1 | tr -d '.') && \
cmake -GNinja -DBUILD_BENCHMARK=ON .. && \
ninja && \
cd ..

Copying run_benchmarks_old.py to a server

scp \
  ~/Downloads/run_benchmarks_old.py \
  myname@myserver:~/rocm-libraries/projects/hipcub

Running develop benchmarks

mkdir results_develop && \
python3 run_benchmarks_old.py \
  --benchmark_dir build_develop/benchmark \
  --benchmark_gpu_architecture $gfx \
  --benchmark_output_dir results_develop \
  --trials 100

Copying develop benchmark results from a server

scp \
  -r \
  myname@myserver:~/rocm-libraries/projects/hipcub/results_develop \
  ~/Downloads

Converting develop JSON results to primbench CSVs

The script gbench2primbench.py converts a directory of old Google Benchmark JSON files to a directory of new primbench CSV files.

--help options

usage: gbench2primbench.py [-h] --project {rocprim,rocrand,hipcub,rocthrust} --noise-threshold-percentage
                           NOISE_THRESHOLD_PERCENTAGE
                           input_dir output_dir

Convert Google Benchmark JSON to primbench CSV format

positional arguments:
  input_dir             Directory containing Google Benchmark JSON files
  output_dir            Output directory for primbench CSV files

options:
  -h, --help            show this help message and exit
  --project {rocprim,rocrand,hipcub,rocthrust}
                        Project name
  --noise-threshold-percentage NOISE_THRESHOLD_PERCENTAGE
                        The noise threshold percentage, past which benchmark specializations are considered
                        to be too noisy

python3 gbench2primbench.py \
  --project hipcub \
  --noise-threshold-percentage 1 \
  results_develop \
  converted

Printing noisy benchmarks

The script print_noisy.py takes the path to a directory of primbench CSV files, and prints CSV file names that contain specializations that timed out due to noise.

--help options

usage: print_noisy.py [-h] csv_dir

Summarize noise from CSV files

positional arguments:
  csv_dir     Path to directory containing CSV files

options:
  -h, --help  show this help message and exit

python3 print_noisy.py results_primbench

Generating benchmark graphs

The script visbench.py takes an old and a new directory of primbench JSON and CSV files and generates an SVG visualization. It automatically matches files between the directories by their filenames, merges the underlying data, and calculates the bytes_per_second speedup ratios.

The resulting graph displays the percentage distribution of these performance speedups across benchmarks. A legend at the top specifies the color mapping for the speedup ratios, categorizing them into 7 discrete buckets (ranging from <0.9× to >1.1×).

Say a unique_by_key benchmark has a large red area comprising 60% of its total bar. This means that 60% of its benchmark specializations fell into the <0.9× speedup bucket. In other words, 60% of those specializations became more than 10% slower.

--help options

Usage: visbench.py [-h] old_dir new_dir output

Positional Arguments:
  old_dir     path to directory containing old primbench .json or .csv files
  new_dir     path to directory containing new primbench .json or .csv files
  output      path to output .svg graph

Options:
  -h, --help  show this help message and exit

python3 visbench.py data/visbench/old/ data/visbench/new/ results/visbench.svg

Generating specialization graphs

The script grapher.py takes primbench JSON and CSV files, and generates a relative or absolute graph.

In the previous step gbench2primbench.py was used to convert old Google Benchmark JSON files to primbench CSV files.

Absolute graphs can be generated by passing --absolute to grapher.py.

--help options

Usage: grapher.py [-h] --output OUTPUT [--algo ALGO] [--arch ARCH] [--filter FILTER] [--absolute] input_files [input_files ...]

Positional Arguments:
  input_files      paths to input .json or .csv primbench files

Options:
  -h, --help       show this help message and exit
  --output OUTPUT  path to output .svg graph (default: None)
  --algo ALGO      algorithm name for the chart title (required if only CSV files are passed) (default: None)
  --arch ARCH      GPU arch name for the chart title (required if only CSV files are passed) (default: None)
  --filter FILTER  regex pattern of specializations to include (default: None)
  --absolute       perform absolute instead of relative comparison (default: False)

python3 -m pip install numpy pygal pandas rich_argparse scipy && \
for csv in converted/*.csv; do
  b=$(basename "$csv" .csv)
  python3 grapher.py \
    --output "graphs/$b.svg" \
    "converted/$b.csv" \
    "results_primbench/$b.json"
done

Test Result

No meaningful performance regressions and noise timeouts were observed.

I ran all 35 hipCUB benchmarks on an MI300X GPU.

Noisy Benchmarks

Five benchmarks still have more than 1% noise: block_reduce, block_scan, device_segmented_radix_sort, device_segmented_sort, and warp_merge_sort. These have been passed settings.noise_tolerance_percent values of 2 and 3 to prevent timeouts.

Paul Kliemann is working on a primbench PR that will significantly lower the noise of benchmark specializations, by calculating entropy instead. rocRAND will automatically benefit from it.

Benchmark Graphs

See the Generating benchmark graphs reproduction step for how to generate benchmark graphs.

primbench its GPU Cache Clearing header explains the difference between the "hot" and "cold" graphs:

primbench clears the GPU cache before each batch to reduce noise and ensure consistent timings. This simulates a "cold run" by preventing leftover data from previous kernel executions from affecting results.

• The --hot flag skips cache clearing, allowing a "hot cache" scenario where data is reused between batches.
• The default cache size is 256 MiB, but PRIMBENCH_GPU_CACHE_SIZE can be overridden at compile time to match your GPU's cache size.

Cold mode is what is getting merged, which by its very nature lowers the throughput of benchmarks. Hot mode graphs are included, as they prove that no bugs were introduced when porting benchmarks from Google Benchmark to primbench.

The specialization graphs from grapher.py are primarily intended for investigating specific regressions.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[umfranzw]

I've got a TheRock PR up here to make amd_smi available for hipCUB: ROCm/TheRock#6100.
It'll need to be merged before CI will pass here.

[Naraenda]

Rebased to resolve conflicts

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
🌿 Branch Name	✅ Pass	—
📝 PR Title/Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

[therock-pr-bot]

🎉 All checks passed! This PR is ready for review.

[NguyenNhuDi]

Hi @Naraenda, @MyNameIsTrez, is this PR ready for review?

[MyNameIsTrez]

Hi @NguyenNhuDi, yes it is. I can't attach the graphs to this PR, but there are basically no regressions and no noise timeouts. There are a handful of small TODOs in the PR's description for missing explanations that I will resolve tomorrow.

[MyNameIsTrez]

Hi @NguyenNhuDi, I have resolved all TODOs in this PR.

I'll clean up its commit history tomorrow by squashing a lot of its commits.

If anyone has any questions, feel free to leave a comment and I will respond.

[MyNameIsTrez]

@NguyenNhuDi I've squashed all commits into one and split that back into 8 commits. Those 8 commits are listed under the Technical Details header. This should make the PR's code changes significantly easier to review.

[NguyenNhuDi]

Looks good to me! Should be good to merge after the merge conflict is resolved and CI is finished running,
I'll keep an eye out on CI.

[Naraenda]

Resolved conflicts and queued CI.

[NguyenNhuDi]

lgtm again

[NguyenNhuDi]

This PR depends on the CCCL changes landing first. Since we're likely merging the copy (#9931) instead of #4079, I'll rebase this PR onto #9931 rather than #4079 once #9931 is done