[BoundsSafety][Compiler-rt] Add lit test suite for -fbounds-safety runtime behavior#12908
Open
delcypher wants to merge 1 commit intoswiftlang:stable/21.xfrom
Open
Conversation
…ntime behavior
This introduces a new test suite under `compiler-rt/test/bounds_safety/`
that verifies `-fbounds-safety` trapping behavior at runtime.
Currently the test configurations only support macOS hosts. This can be extended in the
future. This patch only includes a single test case (that becomes 20 due to
multiple configurations being tried) that is meant to illustrate how testing works.
We can extend the test suite in later patches.
The test suite is designed to exercise multiple dimensions of the bounds-safety
feature:
**Test dimensions (each test runs across all combinations):**
* Optimization level: `-O0` (unopt) and `-O2` (opt)
* Trap kind: `unique-traps`, `merged-traps` (opt-only), and `soft-traps`
* Run mode: `direct` (just run the binary) and `debugger` (run under
LLDB to verify trap location and trap reason)
Each of these dimmensions is handled by generating a different Lit test suite
(a directory under `compiler-rt/test/bounds_safety`) for each configuration.
The `direct` mode exists to perform testing when LLDB isn't available.
**CMake integration:**
* Detects LLDB availability and verifies that the Python interpreter
used by lit can import the `lldb` module (avoids confusing
`ImportError` at test time from Python version mismatches).
* Provides `COMPILER_RT_BOUNDS_SAFETY_USE_LLDB` option that defaults to
ON when LLDB is detected and Python-compatible. It defaults to off a
usable LLDB cannot be found. If set to `On` by the user the CMake
configure will error if a usable LLDB is not found.
* Generates a separate lit site config for each
`(arch, opt_level, trap_kind, run_mode)` combination.
* Host (osx) test suites are aggregated under a new `check-bounds-safety` target.
* A new `COMPILER_RT_TEST_BOUNDS_SAFETY` CMake option is added that
allows disabling the tests.
**Test scripts:**
* `expect_trap.py` — Verifies a binary traps as expected. In direct
mode, it checks that the process is killed by a signal (hard traps) or
emits a marker to stderr (soft traps). In debugger mode, it parses
verify comments from the source file (e.g.
`// expect-trap@+1{indexing above upper bound in 'ptr[idx]'}`) and
checks the trap location, message, and instruction encoding. For
merged traps, only the function name is verified since line info is
lost. These comments are inspired by clang's diagnostic verification
syntax.
* `expect_no_trap.py` — Verifies a binary runs to completion without
trapping. In soft-trap mode, also checks that no soft trap marker
appears in stderr.
* `lldb_utils.py` — Shared LLDB helpers including a context manager
(`LLDBProcessContextManager`) for target creation, process launch, and
cleanup.
**Soft trap support:**
* `soft_trap_runtime_impl.h` provides a `__bounds_safety_soft_trap`
implementation that prints a marker to stderr.
* When the LLDB BoundsSafety instrumentation plugin is not available
(older LLDB versions), `expect_trap.py` falls back to setting a
manual breakpoint and extracting the trap message from the
`__clang_trap_msg$` frame in the call stack.
**Sample test (`bidi_indexable/lower_upper_check.c`):**
This patch includes a sample test case to demonstrate how the test suite
is used. Ir exercises `__bidi_indexable` pointer indexing with lower
bound, upper bound, and in-bounds accesses.
**Alternatives:**
One alternative location to compiler-rt I considered was
`cross-project-tests` which was set up inside llvm-project to handle the
case of testing against multiple projects. That is relevant in this case
because we need clang and ideally lldb to test. The problem there is
that even though there is support for runtime testing (e.g.
`cross-project-tests/debuginfo-tests/dexter-tests/asan.c`) there isn't
proper support there for depending on the LLVM_ENABLE_RUNTIMES style of
building compiler-rt (which is required for proper compiler
bootstrapping). Right now putting the tests inside `compiler-rt` itself
is just easier.
Assisted-by: Claude Code
rdar://175895670
(cherry picked from commit 300cd7f)
Author
|
@swift-ci test |
Author
|
@swift-ci test llvm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This introduces a new test suite under
compiler-rt/test/bounds_safety/that verifies-fbounds-safetytrapping behavior at runtime.Currently the test configurations only support macOS hosts. This can be extended in the future. This patch only includes a single test case (that becomes 20 due to multiple configurations being tried) that is meant to illustrate how testing works. We can extend the test suite in later patches.
The test suite is designed to exercise multiple dimensions of the bounds-safety feature:
Test dimensions (each test runs across all combinations):
-O0(unopt) and-O2(opt)unique-traps,merged-traps(opt-only), andsoft-trapsdirect(just run the binary) anddebugger(run under LLDB to verify trap location and trap reason)Each of these dimmensions is handled by generating a different Lit test suite (a directory under
compiler-rt/test/bounds_safety) for each configuration.The
directmode exists to perform testing when LLDB isn't available.CMake integration:
lldbmodule (avoids confusingImportErrorat test time from Python version mismatches).COMPILER_RT_BOUNDS_SAFETY_USE_LLDBoption that defaults to ON when LLDB is detected and Python-compatible. It defaults to off a usable LLDB cannot be found. If set toOnby the user the CMake configure will error if a usable LLDB is not found.(arch, opt_level, trap_kind, run_mode)combination.check-bounds-safetytarget.COMPILER_RT_TEST_BOUNDS_SAFETYCMake option is added that allows disabling the tests.Test scripts:
expect_trap.py— Verifies a binary traps as expected. In direct mode, it checks that the process is killed by a signal (hard traps) or emits a marker to stderr (soft traps). In debugger mode, it parses verify comments from the source file (e.g.// expect-trap@+1{indexing above upper bound in 'ptr[idx]'}) and checks the trap location, message, and instruction encoding. For merged traps, only the function name is verified since line info is lost. These comments are inspired by clang's diagnostic verification syntax.expect_no_trap.py— Verifies a binary runs to completion without trapping. In soft-trap mode, also checks that no soft trap marker appears in stderr.lldb_utils.py— Shared LLDB helpers including a context manager (LLDBProcessContextManager) for target creation, process launch, and cleanup.Soft trap support:
soft_trap_runtime_impl.hprovides a__bounds_safety_soft_trapimplementation that prints a marker to stderr.expect_trap.pyfalls back to setting a manual breakpoint and extracting the trap message from the__clang_trap_msg$frame in the call stack.Sample test (
bidi_indexable/lower_upper_check.c):This patch includes a sample test case to demonstrate how the test suite is used. Ir exercises
__bidi_indexablepointer indexing with lower bound, upper bound, and in-bounds accesses.Alternatives:
One alternative location to compiler-rt I considered was
cross-project-testswhich was set up inside llvm-project to handle the case of testing against multiple projects. That is relevant in this case because we need clang and ideally lldb to test. The problem there is that even though there is support for runtime testing (e.g.cross-project-tests/debuginfo-tests/dexter-tests/asan.c) there isn't proper support there for depending on the LLVM_ENABLE_RUNTIMES style of building compiler-rt (which is required for proper compiler bootstrapping). Right now putting the tests insidecompiler-rtitself is just easier.Assisted-by: Claude Code
rdar://175895670
(cherry picked from commit 300cd7f)