Github Builds

New CI Testing Configuration

Compiler/Unit Test	Benchmark	Tensor Example	Tensor	openCL	uBLAS	Coverage
Apple Clang Darwin 11 C++11	✔	❌	❌	❌	✔	❌
Apple Clang Darwin 11 C++17	✔	✔	✔	❌	✔	❌
Apple Clang Darwin 11 C++2a	✔	✔	✔	❌	✔	❌
Windows MSVC 14.16 C++11	✔	❌	❌	❌	✔	❌
Windows MSVC 14.25 C++11	✔	❌	❌	❌	✔	❌
Windows MSVC 14.25 C++17	✔	✔	✔	Disabled	✔	❌
Windows MSVC 14.25 C++Latest	✔	✔	✔	❌	✔	❌
Linux Clang 6 C++11	✔	❌	❌	❌	✔	❌
Linux Clang 8 C++11	✔	❌	❌	❌	✔	❌
Linux Clang 9 C++11	✔	❌	❌	❌	✔	❌
Linux Clang 10 C++11	✔	❌	❌	❌	✔	❌
Linux Clang 6.0 C++17	✔	✔	✔	❌	✔	❌
Linux Clang 8 C++17	✔	✔	✔	❌	✔	❌
Linux Clang 9 C++17	✔	✔	✔	❌	✔	❌
Linux Clang 10 C++17	✔	✔	✔	❌	✔	❌
Linux Clang 10 C++2a	✔	✔	✔	❌	✔	❌
Linux GCC 7 C++11	✔	❌	❌	❌	✔	❌
Linux GCC 8 C++11	✔	❌	❌	❌	✔	❌
Linux GCC 9 C++11	✔	❌	❌	❌	✔	❌
Linux GCC 10 C++11	✔	❌	❌	❌	✔	❌
Linux GCC 7 C++17	✔	✔	✔	Disabled	✔	❌
Linux GCC 8 C++17	✔	✔	✔	❌	✔	❌
Linux GCC 9 C++17	✔	✔	✔	❌	✔	✔
Linux GCC 10 C++17	✔	✔	✔	❌	✔	❌
Linux GCC 10 C++2a	✔	✔	✔	❌	✔	❌

✔ : Testing and Stabilized. Should be passing always.

❌ : Not testing

Disabled: Testing has been explicitly disabled due to instability.

Compile Flags and Build Information

OS	Toolchain	Compiler Flags
Linux Ubuntu 20.04	GCC	-O0
Linux Ubuntu 20.04	Clang	-O3
Windows 10	MSVC	No Special Flags
MacOS Catalina	Clang	No Special Flags
UB Sanitizer	GCC and Clang	"-g -fsanitize=undefined"
TH Sanitizer	GCC and Clang	"-g -fsanitize=thread -O2"
ADD Sanitizer	GCC and Clang	"-g -fsanitize=address -fno-omit-frame-pointer"
Clang Tidy	Clang-Tidy-10	Configuration File
Clang Format	Clang-Format-10	Configuration File
Codecov	gcov 9.3.0	No Branch Coverage & only tensor module

Conclusion

• In C++11 CI will not build Tensor Examples, Tensor Tests.
• Coverage is only being reported by Linux GCC-7 C++17.
• OpenCL testing is being done by MSVC 14.25 C++17 and GCC-7 C++17. Both of them are unstable in testing OpenCL. Due to poor setup of openCL toolkit or some other reasons, the CI sometimes fails and sometimes succeeds randomly.
• All CI will test uBLAS and Benchmark.
• All CI above C++11 will test the Tensor Example and Tensor Unit tests.

Time and Performance

Depending upon the number of tests a CI is running, the time of completion varies across the job, It is also worth noticing that openCL test usually take longer than expected, taking over 15 min in some cases due to the amount of openCL setup required prior to building the test.

Generally, the complete test suit with all CI configuration will complete in under 15 min, most of the above jobs run in parallel and depending upon the number of runners available in the GitHub Runner pool, the amount of parallelism may change.

Scope of Improvements

• Using cache to Cache the compiler intermediate builds.
• Opting more reliable way to set up openCL on CI quickly. Cem is likely to replace openCL with openMP.

UPDATE (1-June-2020)

I have disabled all OpenCL tests, this speeds the build process and now CI is very reliable.