Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve and generalize specialization testing utilities. #766

Open
jswrenn opened this issue Sep 28, 2023 · 5 comments
Open

Improve and generalize specialization testing utilities. #766

jswrenn opened this issue Sep 28, 2023 · 5 comments

Comments

@jswrenn
Copy link
Member

jswrenn commented Sep 28, 2023

This issue tracks two complimentary directions for improving our specialization tests:

  • @Philippe-Cholet noted that our specialization tests do not currently test partially-consumed iterators.
  • We should extend our specialization testing pattern to benchmarking, so both each specialized (and unspecialized) method are tested in a benchmark group.
@Philippe-Cholet
Copy link
Member

Philippe-Cholet commented Sep 29, 2023

I'm definitely gonna work on improving specialization tests (as mentioned) and benchmarks (at some point).

@Philippe-Cholet
Copy link
Member

Philippe-Cholet commented Sep 29, 2023

After working on combinations & co which produce a lot of elements, a "struggle" I sometimes encounter when writing quickcheck tests is to have small enough iterators on which apply our itertools adaptors (otherwise tests would be too slow).
The current iterators we rely on are mostly the custom Iter<u8, Inexact(default) or Exact> and Vec<u8>. We can call .take(...) on Iter (but eventually lose the unfused element at the end) and truncate vectors.
But it surely does not reduce the numbers of tests. Let's say it tests against vectors of lengths 0..=128 and we truncate each to 10 inside the test then it tests against a lot of iterators of length 10, mostly unnecessarily. I think the problem is that the truncation is done too late to reduce the number of tests.

So I think it would be valuable to have a way to generate small iterators, especially for specialization tests which I expect to be slower if we want to test against more than fresh iterators.
Even without const generics Iter<T, const SIZE_LIMIT: usize>, I'm sure we could have Iter<T, LIMIT=NoLimit> or Iter<T, Max10>. And similarly wraps vectors into struct LittleVec<T, LIMIT=Max10>(Vec<T>);.
Maybe LittleVec would be enough.
Maybe you have a better idea.

This is merely a though I had, I did not work on it yet.

@jswrenn
Copy link
Member Author

jswrenn commented Sep 29, 2023

The simplest thing we can do, right now, is to return, rather than truncate. That won't totally discard the test, but it will avoid most of the computational overhead of running it.

To totally discard a test, see this guidance in the quickcheck readme: https://github.com/BurntSushi/quickcheck#discarding-test-results-or-properties-are-polymorphic

@Philippe-Cholet
Copy link
Member

Thanks! ...I guess I simply forgot about TestResult, autocompletion would have then saved me. 😥

@Philippe-Cholet
Copy link
Member

If #786 is not enough to extend our specialization benchmarking, we'll make another PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants