-
Notifications
You must be signed in to change notification settings - Fork 317
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature request] Optional caching of tests #1703
Comments
Our general advice for this is to break your tests up into files corresponding to your R code, and then just run the test file related to the R file you're currently working on using |
Thanks for your response. My feeling is that, while doing things as you advise is ideal (not least because it forces you to structure your code thoughtfully), there's a set of problems which caching would solve, mainly because a lot of code isn't structured well and doesn't lend itself to the ideal approach. E.g. someone inheriting a big, poorly written package might find that caching would substantially reduce the cognitive load of refactoring the codebase, particularly if many tests are lengthy and many functions need to be moved to new scripts. I can think of a few examples in my own job where this would be the case. This basically boils down to the fact that, if you're working on a change which affects more than one file, running Another nice thing about caching would be that |
If your tests take too long to run, I think you are better off refactoring them to make them faster. IMO doing anything automatically with caching is extremely likely to be a net negative, because you will get intermittent problems due to not getting the cache invalidation 100% correct. |
FWIW I use the workflow you recommend, and I was using it at the exact time I laid down this idea. Regarding implementation Jim shared on twitter that {covr} might do a fair chunk of the work already : https://covr.r-lib.org/reference/covr.record_tests.html |
Sorry not to be clear, I really don't think this is a good idea and it's not something that I think is a good fit for testthat. I'd suggest implementing it in another package or explicitly caching your tests yourself. |
Absolutely no worries. Thanks for your comments/consideration 🙂 |
I agree with the general sentiment that each r-lib/pillar@ For projects where this isn't feasible right now, I have a small package that reruns only failing tests: https://github.com/krlmlr/lazytest. |
Why cache tests?
If you're working on a big package with a test suite that takes a long time to run, having an option to only run tests that call functions (directly or indirectly) where code has changed could potentially bring lengthy calls to
test()
down to only a few seconds.How would this work?
You'd need to check whether any functions that are directly or indirectly called during a test have changed since the last time it was run. I can think of two possible implementations:
Method 1: Use code parsing
You would need to parse each function called during the test to get a dependency tree, then inspect this for changes. This opens a can of worms because extracting function calls from R code is hard. E.g. how do you generalise the detection of the function
foo
inlapply(foo, 1:10)
?Method 2: Modify functions before running
test_that()
This would work something like this:
When
test_that()
gets run, every function in the package is modified so that it records its (original) body whenever it gets called:At the end of the test,
called_funs
will contain the bodies of all the functions in myPackage which have been called, directly or indirectly, during the test. (NB, you'd also want to record function arguments but that's omitted for the sake of brevity).This list of functions definitions can then be checked for changes each time the test gets called, e.g. using snapshotting. If the snapshot passes (and the code for the test itself hasn't changed), the full test doesn't need to be run.
Possible issues
I can see two potential issues here:
Modifying the definition for
foo
as outlined above breaks something. I can't think of any reason why this would happen, but there's a lot of weird R code out there. This could be handled by making the caching opt-in.Functions don't change superficially, but some dependency, e.g. some C++ code, does. Again, I think this could be handled by making the caching opt-in, or maybe opt-out for specific tests.
NB, this issue was prompted by a discussion on Twitter with @moodymudskipper.
As ever, thanks for the hard work on this amazing package.
The text was updated successfully, but these errors were encountered: