Skip to content

mercury: add variant to build perf tests #2902

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
alalazo merged 5 commits into from
Jan 19, 2026
Merged

mercury: add variant to build perf tests #2902

alalazo merged 5 commits into from
Jan 19, 2026

Conversation

@nhanford
Copy link
Contributor

@nhanford nhanford commented Dec 29, 2025

Hello @soumagne,

This PR adds a variant that builds and persistently installs the performance tests for Mercury.
Please let me know in which versions these CMake flags were introduced.

Preliminary results from a system similar to El Capitan at LLNL (200Gbps/HSA):

na_bw_get -c ofi -p cxi -b -l 1000
...
# [859097.930837] mercury->op [warning] /var/tmp/nhanford/spack-stage/spack-stage-mercury-master-ks4eadvpsbbqrwjxzxkebaxqymczvpry/spack-src/src/na/na_ofi.c:6950 na_ofi_cq_readerr() fi_cq_readerr() got err: 5 (Input/output error), prov_errno: 18 (ENTRY_NOT_FOUND)
# [859097.930842] mercury->op [error] /var/tmp/nhanford/spack-stage/spack-stage-mercury-master-ks4eadvpsbbqrwjxzxkebaxqymczvpry/spack-src/src/na/na_ofi.c:7137 na_ofi_cq_process_error() error event on operation ID 0x555555667670 (NA_CB_GET), fi_readmsg(iov_count=1, desc[0]=0x555555608680, msg_iov[0].iov_base=0x5555976f1000, msg_iov[0].iov_len=16384, addr=1, rma_iov_count=1, rma_iov[0].addr=0x3e000000, rma_iov[0].len=16384, rma_iov[0].key=0x900006b999e70000, context=0x5555556678b0, data=0) failed, rc: 5 (Input/output error)
16384                     15224.32                    1.03
32768                     22036.48                    1.42
65536                     22591.44                    2.77
131072                    22581.05                    5.54
262144                    23008.03                   10.87
524288                    23060.44                   21.68
1048576                   23117.53                   43.26
2097152                   23139.96                   86.43
4194304                   23152.97                  172.76
8388608                   23159.87                  345.43
16777216                  23163.12                  690.75

Thanks,
Nate

@spackbot-triage spackbot-triage bot requested a review from soumagne December 29, 2025 18:43
alalazo
alalazo previously approved these changes Jan 7, 2026
View reviewed changes
@alalazo alalazo self-assigned this Jan 7, 2026
@soumagne
Copy link
Contributor

soumagne commented Jan 9, 2026

@nhanford Apologies for the slow response. BUILD_TESTING_PERF was added in 2.3.0. You can add it next to that:

        if "@2.3.0:" in spec:
            cmake_args.append(define("BUILD_TESTING_UNIT", self.run_tests))

BUILD_TESTING has always been there as far as I recall. For some reason I had the impression that this was already supported in the spack recipe...

@soumagne
Copy link
Contributor

soumagne commented Jan 9, 2026

Also unrelated to that, @nhanford the error that you're getting when running that perf test with cxi is something that we've been investigating. Could you please let me know if you have seen that issue frequently when running that particular benchmark ? Thanks.

@nhanford
Copy link
Contributor Author

nhanford commented Jan 9, 2026

@soumagne Thanks for the info. This version should be a lot cleaner. Unfortunately I was not able to test it on my system because the build failed for me due to some CMake issues, but this should work.
Yes I observed that failure for smaller message sizes persistently across many tests. The system under test is identical to LLNL El Capitan (Cray EX, Slingshot, AMD CPU/APU) except it uses Slingshot Host Software 13 and CXI VNIs set using Flux.

@nhanford nhanford mentioned this pull request Jan 9, 2026
Closed
soumagne
soumagne reviewed Jan 9, 2026
View reviewed changes
@soumagne
Copy link
Contributor

soumagne commented Jan 9, 2026 •
edited
Loading

@soumagne Thanks for the info. This version should be a lot cleaner. Unfortunately I was not able to test it on my system because the build failed for me due to some CMake issues, but this should work. Yes I observed that failure for smaller message sizes persistently across many tests. The system under test is identical to LLNL El Capitan (Cray EX, Slingshot, AMD CPU/APU) except it uses Slingshot Host Software 13 and CXI VNIs set using Flux.

thanks this is very useful information. If you are able to reproduce this error consistently, could you please run with the following env vars set HG_LOG_LEVEL=warn HG_LOG_SUBSYS=hg,na,libfabric and file an issue in mercury's github ? we can follow up there. Thanks!

@spackbot-triage spackbot-triage bot requested a review from soumagne January 12, 2026 18:05
@nhanford
Copy link
Contributor Author

nhanford commented Jan 14, 2026

@soumagne Thanks for the info. This version should be a lot cleaner. Unfortunately I was not able to test it on my system because the build failed for me due to some CMake issues, but this should work. Yes I observed that failure for smaller message sizes persistently across many tests. The system under test is identical to LLNL El Capitan (Cray EX, Slingshot, AMD CPU/APU) except it uses Slingshot Host Software 13 and CXI VNIs set using Flux.

thanks this is very useful information. If you are able to reproduce this error consistently, could you please run with the following env vars set HG_LOG_LEVEL=warn HG_LOG_SUBSYS=hg,na,libfabric and file an issue in mercury's github ? we can follow up there. Thanks!

Since these errors appear not to affect performance and only seem to appear on our test system, I'm going to try to run this down locally and see how far I can get, and then I will file a bug if the issue persists. Thanks!

@soumagne
Copy link
Contributor

soumagne commented Jan 14, 2026

@soumagne Thanks for the info. This version should be a lot cleaner. Unfortunately I was not able to test it on my system because the build failed for me due to some CMake issues, but this should work. Yes I observed that failure for smaller message sizes persistently across many tests. The system under test is identical to LLNL El Capitan (Cray EX, Slingshot, AMD CPU/APU) except it uses Slingshot Host Software 13 and CXI VNIs set using Flux.

thanks this is very useful information. If you are able to reproduce this error consistently, could you please run with the following env vars set HG_LOG_LEVEL=warn HG_LOG_SUBSYS=hg,na,libfabric and file an issue in mercury's github ? we can follow up there. Thanks!

Since these errors appear not to affect performance and only seem to appear on our test system, I'm going to try to run this down locally and see how far I can get, and then I will file a bug if the issue persists. Thanks!

Thanks! that will be much appreciated as we've been trying to narrow this down...

@soumagne soumagne requested a review from alalazo January 15, 2026 16:15
@alalazo alalazo merged commit 2ea40cb into spack:develop Jan 19, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alalazo alalazo alalazo approved these changes

+1 more reviewer

@soumagne soumagne soumagne approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@alalazo alalazo

Labels

new-variant update-package

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nhanford @soumagne @alalazo