Skip to content

Conversation

@maxammann
Copy link
Contributor

@maxammann maxammann commented Oct 20, 2023

This adds experimental support for generating coverage using LLVM and cargo-llvm-cov. Note that replacing cargo-llvm-cov would be simple. It might simplify stuff even as cargo-llvm-cov basically just invokes llvm-profdata and llvm-cov.

How to use

Suppose you have a setup like in the examples directory of the repository. Then:

  1. Install cargo-llvm-cov: cargo install cargo-llvm-cov
  2. Install LLVM tools (they might get installed by cargo-llvm-cov automatically, we need to test that): rustup toolchain install nightly --component llvm-tools-preview
  3. Run cargo test
  4. Optionally, fuzz the project.
  5. Execute cargo-test-fuzz test-fuzz target --no-instrumentation --replay corpus --coverage

Coverage is generated as HTML in target/llvm-cov/html.

image

TODO:

  • Should --coverage imply --no-instrumentation?
  • Should --coverage be compatible with fuzzing? (probably not, even though it might just work)
  • Make sure that llvm-tools-preview and cargo-llvm-cov get installed or promt for installation.
  • On macOS I get the following warning during generation: warning: 4 functions have mismatched data
  • During previous experiments I got such coverage data:
    image
    I suspect this is due to the fact that test-fuzz injects code into the function-to-test here:
    #write_concretizations_and_args

    Either move that code somewhere else, or put it into a function, ad a no_coverage instruction and call that from the target function.

Details

The follwing commands are executes to generate coverage. We should validate them:

     Running `/Users/max/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/llvm-profdata merge -sparse -f /Users/max/projects/examples/target/examples-profraw-list -o /Users/max/projects/examples/target/examples.profdata`
     Running `/Users/max/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/llvm-cov show -format=html -instr-profile=/Users/max/projects/examples/target/examples.profdata -object target/debug/deps/alloc-34551e9cc1ff6b3f -object target/debug/deps/arc-bf8fdeaeed680770 -object target/debug/deps/assert-ba6618cd737ff666 -object target/debug/deps/associated_type-7ed59268473251eb -object target/debug/deps/auto_concretize_0-736514e466198bc3 -object target/debug/deps/auto_generate-7287f7108f5b00a9 -object target/debug/deps/conversion-c5a93588844a6594 -object target/debug/deps/debug-ff5617e8189a0951 -object target/debug/deps/default-79bdc830b898ec0f -object target/debug/deps/from-2bc06bf9dd96fea9 -object target/debug/deps/generic-23602c4ce1352038 -object target/debug/deps/hello_world-594474f770d22b32 -object target/debug/deps/hello_world-cff0c92d5749ded3 -object target/debug/deps/lifetime-f34199c00924285b -object target/debug/deps/manual_leak-7a7f8bfdea2a3f10 -object target/debug/deps/mut_ref-295c3cd2737a4d53 -object target/debug/deps/parse_duration-d93f2ce3d20bd782 -object target/debug/deps/qwerty-72231f74eb29ea0a -object target/debug/deps/rename-503984417115eb3f -object target/debug/deps/return_type-3b4f1b217dc76985 -object target/debug/deps/serde-26f50433033e3e5d -object target/debug/deps/test_fuzz_impl-2a426bdf99ecc53a -object target/debug/deps/unserde-4a2d01352b369939 -object target/debug/hello-world -ignore-filename-regex 'test-fuzz|/rustc/([0-9a-f]+|[0-9]+\.[0-9]+\.[0-9]+)/|^/Users/max/projects/examples(/.*)?/(tests|examples|benches)/|^/Users/max/projects/examples/target($|/)|^/Users/max/\.cargo/(registry|git)/|^/Users/max/\.rustup/toolchains($|/)' -show-instantiations=false -show-line-counts-or-regions -show-expansions -show-branches=count -Xdemangler=/Users/max/.cargo/bin/cargo-llvm-cov -Xdemangler=llvm-cov -Xdemangler=demangle -output-dir=/Users/max/projects/examples/target/llvm-cov/html`

fixes #293

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Max Ammann seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add tooling for generating coverage

3 participants