Skip to content

[BoundsSafety][Compiler-rt] Add lit test suite for -fbounds-safety runtime behavior#12908

Open
delcypher wants to merge 1 commit intoswiftlang:stable/21.xfrom
delcypher:dliew/radar-175895670-cherry-pick-stable-21.x
Open

[BoundsSafety][Compiler-rt] Add lit test suite for -fbounds-safety runtime behavior#12908
delcypher wants to merge 1 commit intoswiftlang:stable/21.xfrom
delcypher:dliew/radar-175895670-cherry-pick-stable-21.x

Conversation

@delcypher
Copy link
Copy Markdown

This introduces a new test suite under compiler-rt/test/bounds_safety/ that verifies -fbounds-safety trapping behavior at runtime.

Currently the test configurations only support macOS hosts. This can be extended in the future. This patch only includes a single test case (that becomes 20 due to multiple configurations being tried) that is meant to illustrate how testing works. We can extend the test suite in later patches.

The test suite is designed to exercise multiple dimensions of the bounds-safety feature:

Test dimensions (each test runs across all combinations):

  • Optimization level: -O0 (unopt) and -O2 (opt)
  • Trap kind: unique-traps, merged-traps (opt-only), and soft-traps
  • Run mode: direct (just run the binary) and debugger (run under LLDB to verify trap location and trap reason)

Each of these dimmensions is handled by generating a different Lit test suite (a directory under compiler-rt/test/bounds_safety) for each configuration.

The direct mode exists to perform testing when LLDB isn't available.

CMake integration:

  • Detects LLDB availability and verifies that the Python interpreter used by lit can import the lldb module (avoids confusing ImportError at test time from Python version mismatches).
  • Provides COMPILER_RT_BOUNDS_SAFETY_USE_LLDB option that defaults to ON when LLDB is detected and Python-compatible. It defaults to off a usable LLDB cannot be found. If set to On by the user the CMake configure will error if a usable LLDB is not found.
  • Generates a separate lit site config for each (arch, opt_level, trap_kind, run_mode) combination.
  • Host (osx) test suites are aggregated under a new check-bounds-safety target.
  • A new COMPILER_RT_TEST_BOUNDS_SAFETY CMake option is added that allows disabling the tests.

Test scripts:

  • expect_trap.py — Verifies a binary traps as expected. In direct mode, it checks that the process is killed by a signal (hard traps) or emits a marker to stderr (soft traps). In debugger mode, it parses verify comments from the source file (e.g. // expect-trap@+1{indexing above upper bound in 'ptr[idx]'}) and checks the trap location, message, and instruction encoding. For merged traps, only the function name is verified since line info is lost. These comments are inspired by clang's diagnostic verification syntax.
  • expect_no_trap.py — Verifies a binary runs to completion without trapping. In soft-trap mode, also checks that no soft trap marker appears in stderr.
  • lldb_utils.py — Shared LLDB helpers including a context manager (LLDBProcessContextManager) for target creation, process launch, and cleanup.

Soft trap support:

  • soft_trap_runtime_impl.h provides a __bounds_safety_soft_trap implementation that prints a marker to stderr.
  • When the LLDB BoundsSafety instrumentation plugin is not available (older LLDB versions), expect_trap.py falls back to setting a manual breakpoint and extracting the trap message from the __clang_trap_msg$ frame in the call stack.

Sample test (bidi_indexable/lower_upper_check.c):

This patch includes a sample test case to demonstrate how the test suite is used. Ir exercises __bidi_indexable pointer indexing with lower bound, upper bound, and in-bounds accesses.

Alternatives:

One alternative location to compiler-rt I considered was cross-project-tests which was set up inside llvm-project to handle the case of testing against multiple projects. That is relevant in this case because we need clang and ideally lldb to test. The problem there is that even though there is support for runtime testing (e.g. cross-project-tests/debuginfo-tests/dexter-tests/asan.c) there isn't proper support there for depending on the LLVM_ENABLE_RUNTIMES style of building compiler-rt (which is required for proper compiler bootstrapping). Right now putting the tests inside compiler-rt itself is just easier.

Assisted-by: Claude Code

rdar://175895670
(cherry picked from commit 300cd7f)

…ntime behavior

This introduces a new test suite under `compiler-rt/test/bounds_safety/`
that verifies `-fbounds-safety` trapping behavior at runtime.

Currently the test configurations only support macOS hosts. This can be extended in the
future.  This patch only includes a single test case (that becomes 20 due to
multiple configurations being tried) that is meant to illustrate how testing works.
We can extend the test suite in later patches.

The test suite is designed to exercise multiple dimensions of the bounds-safety
feature:

**Test dimensions (each test runs across all combinations):**

* Optimization level: `-O0` (unopt) and `-O2` (opt)
* Trap kind: `unique-traps`, `merged-traps` (opt-only), and `soft-traps`
* Run mode: `direct` (just run the binary) and `debugger` (run under
	LLDB to verify trap location and trap reason)

Each of these dimmensions is handled by generating a different Lit test suite
(a directory under `compiler-rt/test/bounds_safety`) for each configuration.

The `direct` mode exists to perform testing when LLDB isn't available.

**CMake integration:**

* Detects LLDB availability and verifies that the Python interpreter
	used by lit can import the `lldb` module (avoids confusing
	`ImportError` at test time from Python version mismatches).
* Provides `COMPILER_RT_BOUNDS_SAFETY_USE_LLDB` option that defaults to
	ON when LLDB is detected and Python-compatible. It defaults to off a
  usable LLDB cannot be found. If set to `On` by the user the CMake
  configure will error if a usable LLDB is not found.
* Generates a separate lit site config for each
	`(arch, opt_level, trap_kind, run_mode)` combination.
* Host (osx) test suites are aggregated under a new `check-bounds-safety` target.
* A new `COMPILER_RT_TEST_BOUNDS_SAFETY` CMake option is added that
  allows disabling the tests.

**Test scripts:**

* `expect_trap.py` — Verifies a binary traps as expected. In direct
	mode, it checks that the process is killed by a signal (hard traps) or
	emits a marker to stderr (soft traps). In debugger mode, it parses
	verify comments from the source file (e.g.
	`// expect-trap@+1{indexing above upper bound in 'ptr[idx]'}`) and
	checks the trap location, message, and instruction encoding. For
	merged traps, only the function name is verified since line info is
	lost. These comments are inspired by clang's diagnostic verification
  syntax.
* `expect_no_trap.py` — Verifies a binary runs to completion without
	trapping. In soft-trap mode, also checks that no soft trap marker
	appears in stderr.
* `lldb_utils.py` — Shared LLDB helpers including a context manager
	(`LLDBProcessContextManager`) for target creation, process launch, and
	cleanup.

**Soft trap support:**

* `soft_trap_runtime_impl.h` provides a `__bounds_safety_soft_trap`
	implementation that prints a marker to stderr.
* When the LLDB BoundsSafety instrumentation plugin is not available
	(older LLDB versions), `expect_trap.py` falls back to setting a
	manual breakpoint and extracting the trap message from the
	`__clang_trap_msg$` frame in the call stack.

**Sample test (`bidi_indexable/lower_upper_check.c`):**

This patch includes a sample test case to demonstrate how the test suite
is used. Ir exercises `__bidi_indexable` pointer indexing with lower
bound, upper bound, and in-bounds accesses.

**Alternatives:**

One alternative location to compiler-rt I considered was
`cross-project-tests` which was set up inside llvm-project to handle the
case of testing against multiple projects. That is relevant in this case
because we need clang and ideally lldb to test. The problem there is
that even though there is support for runtime testing (e.g.
`cross-project-tests/debuginfo-tests/dexter-tests/asan.c`) there isn't
proper support there for depending on the LLVM_ENABLE_RUNTIMES style of
building compiler-rt (which is required for proper compiler
bootstrapping). Right now putting the tests inside `compiler-rt` itself
is just easier.

Assisted-by: Claude Code

rdar://175895670
(cherry picked from commit 300cd7f)
@delcypher delcypher requested a review from a team as a code owner May 7, 2026 00:53
@delcypher delcypher added the clang:bounds-safety Issue relating to the experimental -fbounds-safety feature in Clang label May 7, 2026
@delcypher
Copy link
Copy Markdown
Author

@swift-ci test

@delcypher
Copy link
Copy Markdown
Author

@swift-ci test llvm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clang:bounds-safety Issue relating to the experimental -fbounds-safety feature in Clang

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant