Splines improve benchmark

[blegouix]

• Capability to sweep on parameters determined at compile-time: Uniform/NonUniform, Degree, ExecutionSpace and in the futur solver Ginkgo/Lapack.

[blegouix]

@tpadioleau any opinion about this ? It is hacky but helps a lot to make the benchmarks flexible. I did not find yet any way to write it as "nested loops", maybe there is one still.

[blegouix]

I can imagine an improvement of TypeSeq with something like:

template<std::size_t N, typename List>
struct TypeByIndex;

template<std::typename Head, typename... Tail>
struct TypeByIndex<0, ddc::detail::TypeSeq<Head, Tail...>> {
    using type = Head;
};

template<std::size_t N, typename Head, typename... Tail>
struct TypeByIndex<N, ddc::detail::TypeSeq<Head, Tail...>> {
    using type = typename TypeAt<N-1, TypeList<Tail...>>::type;
};

And upon this a nested parameter pack expansions on TypeSeq<Kokkos::DefaultHostExecutionSpace, Kokkos::DefaultExecutionSpace>, TypeSeq<std::false_type, std::true_type> and TypeSeq<3, 4, 5> should be feasible, but it may be a bit too complicated.

[tpadioleau]

@tpadioleau any opinion about this ? It is hacky but helps a lot to make the benchmarks flexible. I did not find yet any way to write it as "nested loops", maybe there is one still.

Don't you think you need different tests for Ginkgo and Lapack ? They have different parameters ? cols_per_chunk_ref and preconditionner_max_block_size_ref do not make sense for Lapack right ?

[tpadioleau]

If you want to select a benchmark based on a runtime parameter could we use a string and store the different benchmarks in a std::map<std::string, benchmark> my_benchmarks ? you would just call my_benchmarks.at("host")

[blegouix]

@tpadioleau any opinion about this ? It is hacky but helps a lot to make the benchmarks flexible. I did not find yet any way to write it as "nested loops", maybe there is one still.

Don't you think you need different tests for Ginkgo and Lapack ? They have different parameters ? cols_per_chunk_ref and preconditionner_max_block_size_ref do not make sense for Lapack right ?

Atm the choice of Backend still has to be manually changed by editing the file. cols_per_chunk_ref and preconditionner_max_block_size_ref do not make sense for Lapack indeed, and this is weird to have them passed as optional arguments of SplineBuilder. But this is the current implementation, we could do something better with a kwArgs struct (similar to what we have in the FFT) but this requires more developments.

About the benchmark, the current way to use it is to comment/uncomment the benchmarks we want to run so we just have to not run the preconditioner or cols_per_chunk benchmark with Lapack.

If you want to select a benchmark based on a runtime parameter could we use a string and store the different benchmarks in a std::map<std::string, benchmark> my_benchmarks ? you would just call my_benchmarks.at("host")

Making a more interfaced benchmark system would be great but requires development. And I think it would not impact the way benchmarks are performed (this would be a software overlay build upon it). Maybe gtest already provides something to do it.

[blegouix]

(my question was more about instanciating all the possible variations of the class we would like to benchmark, combining explicitly compile-time parameters like ExecutionSpace, Degree and Uniformity)

[tpadioleau]

Ok I think I get the problem, would this google benchmark feature https://github.com/google/benchmark/blob/main/docs/user_guide.md#templated-benchmarks-that-take-arguments help you ?

[blegouix]

I don't think so because I think it does not allow to "sweep" on those template parameters (i.e. degree={3,4,5}) so it would be at least less conveniant than my solution (if I understand well the capabilities of this feature).