grcov sometimes takes very long to execute #1013
Comments
|
Looks like the incremental directory is really causing us problems because it includes lots of files for every output target (binaries, the crate itself, and integration tests): https://gist.github.com/jonhoo/ef66c75137b1f9a14c322faa747b8cc1 I also ran perf on |
|
cc @jerel |
|
@marco-c I'm happy to try to help in fixing this, but could use some pointers for how you think this should be handled. For example, would it be okay to just always ignore everything that's in |
|
Thanks for splitting this out and digging into it. What you've documented makes sense with what I've seen in practice. As supporting evidence my workaround solution completes quickly and (I think) does the same work but avoids enumeration of TEST_BIN_PATHS=$( \
cargo test --no-run --message-format=json --tests --lib \
| grep "{\"" \
| jq -r "select(.profile.test == true) | .filenames[]" \
| grep -v dSYM - \
)
rust-profdata merge -sparse ./*.profraw -o combined.profdata
rust-cov export \
--ignore-filename-regex='/.cargo/registry|.cargo/git|/rustc|target/debug' \
--instr-profile=combined.profdata \
--object target/debug/libmy_app.so $TEST_BIN_PATHS \
--format lcov > lcov-combined.info
grcov lcov-combined.info -s . -o ./html -t html --ignore-not-existing --llvmIf I'm not mistaken all of the binaries that grcov needs to load are at the top level of |
|
I think that kind of filtering would end up happening here so we don't even recurse down into that dir: Lines 171 to 196 in 917494a |
|
The challenge in doing it in |
|
So something like an |
|
Ah, wait, no, that's for walking |
|
It's over here we'll need to make a change: Lines 102 to 130 in 917494a |
|
If it's helpful to your exploration I had some uncommitted work from several weeks ago where I was experimenting with accepting IIRC it didn't solve the perf issue by itself (in hindsight probably because it was still walking the |
|
Oh, that's interesting. You should open that as a PR independently of this; I think it's a good change in isolation. |
|
btw, the places you have |
|
I think we have a couple of paths forward:
I don't know which of these paths is better, and we may need the maintainers to chime in on that. I also think it'd be nice if Lines 122 to 132 in 917494a as that would highlight problems in this area much faster. |
|
The ideal solution is probably fixing #535, basically kind of reproducing the way llvm-cov ignores files.
I'm not a fan of special cases, they might silently break things for uncommon use cases.
This would make things more complex for users of grcov. I'd rather we did the right thing by default.
Same as 1.
With this, we'd risk missing some coverage (e.g. from shared objects).
Might work, hopefully it doesn't break anything, but I'm not 100% sure. |
|
Since 5 was easy to try out, I went that path (so no more And here's the 10% of execution time is spent in |
|
So the pipe here is definitely a bottleneck. But the huge amount of cycles spent on |
|
It seems like a fairly straightforward method: But it also appears to do a linear search, which could certainly cause problems for even modestly sized inputs. |
|
One obvious win here would be to run |
|
Just invoking |
|
The project I'm on has 1,400 items in |
|
You can ignore all the |
|
Interestingly, passing the |
This results in a significant speed-up since we can avoid invoking `llvm-cov export` on all the object files in `target/debug/incremental`. See mozilla#1013.
llvm-cov can be fairly slow per invocation, but can safely and easily be parallelized for significant speed-ups when there are many binary artifacts. See mozilla#1013.
llvm-cov can be fairly slow per invocation, but can safely and easily be parallelized for significant speed-ups when there are many binary artifacts. See mozilla#1013.
|
Filed an issue with LLVM about |
llvm-cov can be fairly slow per invocation, but can safely and easily be parallelized for significant speed-ups when there are many binary artifacts. See #1013.
|
Not sure where to ask, but I certainly can help testing things out. Firstly, I'm not really sure (as grcov doesn't output any logs for me) what is going on there. |
|
It might also be worth suggesting that people use a separate profile for coverage on their systems. For example, my debug/deps folder that I was trying to run grcov on has over 40k files in it which made it take ~12m to run! [profile.coverage]
inherits = "test"Adding a coverage profile and doing a fresh test run dropped that number down to ~3k deps files which let grcov do what it needed to do in just over a minute! |
This is extracted from the discussion in #326.
In some projects, some of the time, when I execute
grcov, it takes a very long time to execute, using 100% of one core for several minutes at the time. I executed it like this:I tried running
perfon thegrcovinvocation from shortly after it started until I killed it, and the results were interesting:A run of
strace -f -cwas also instructive:It looks like a lot of time is spent just calling
llvm-cov export, which then spends a lot of its time skipping/ignoring files. And a lot of time is spend on locking somewhere. Looking at what thosellvm-cov exportsoperate on, it looks like it's every file under$X/cb-coverage-target/debug:(note that I killed the process at this point — I assume it would have continued going through all the files)
This... feels like a problem.
The text was updated successfully, but these errors were encountered: