BUIP051: Set up dedicated Continuous Integration Proposer: freetrader Submitted on: 2017-04-15 Status: draft
Get a paid-for dedicated Continuous Integration (CI) service to do daily / nightly builds which cover the full test suite.
This would be an addition to the existing free service provided by Travis.
Right now, BU is running on free instance of Travis CI, which is a great service for open source projects. However, this is limited in the build minutes, which means that BU cannot run the full suite of software tests (which can take up to a few hours to run through depending on hardware).
Instead, it runs a limited subset of quicker tests (a kind of "smoke testing").
Practically, individual developers are limited in the build platforms available to them, and do not always run the extended tests, which results in a lack of awareness of whether changes have broken some parts of the software on some supported platforms.
Indeed, it is difficult if not impossible to currently obtain an accurate picture of the overall project test status (all applicable tests on supported platforms). A dedicated CI service could run the full test suite once a day (testing 'dev' and 'release' branch heads for various build platforms).
Contributing developers need information on how reliably a test is expected to perform - when you develop a feature and the test fails, you want to know whether the test was occasionally known to fail before you started, or whether it's likely due to changes you've made, etc.
Acts as a 'health check' for users who want to run BU. They can look at the test status to gauge whether they are satisfied with the software's health and prepared to take the risk of running it.
Helps the project obtain better quality by making sure they don't break the software accidentally (daily feedback on merged code 'dev' and 'release' branches).
This BUIP does not intend to prescribe a particular service or provider, but strongly recommends going with an existing, well known service with good reputation and public interface.
A 'Startup' Travis instance (described as 'best for small teams') costs $129 / month, i.e. ~$1548 / year, and allows 2 concurrent jobs and unlimited build minutes / repositories / collaborators.
A 'Small Business' Travis instance (described as 'best for growing teams') costs $129 / month, i.e. ~$2988 / year, and allows 5 concurrent jobs and unlimited build minutes / repositories / collaborators.
$3K / year for reliable full-spectrum daily test status information accessible to all project members would be a good investment in the author's view.
The existing free Travis service which we are using to test PRs and merge commits should remain in operation unchanged by this BUIP (this is a recommendation). The free service is useful as-is, a lot of effort has been invested and there is no major benefit to changing it / moving everything to another service.
The daily / nightly tests can also be run quite differently, requiring a different CI configuration. For example, for tests that fail, one could re-run them several times to gauge whether it's a random failure and what the probability of failure is. One could also set up repetitions of passing tests with different execution orders, various random seeds etc. There's a lot that can be done to get more mileage out of the existing tests.
The author is unaware of Bitcoin Core or other Satoshi-based clients publishing full test suite run data to the public. This is certainly not optimal, and a good chance for BU to improve upon current practice and provide a good example.
Developers could set up dedicated test boxes at home / work and feed back test results into a common Git repository.
This would lack the nice interface etc., and would be more costly overall as people might have to acquire dedicated test hardware (although some of this could be obtained through sponsorship or donations).
One could make a separate BUIP for a project to aggregate this "ragtag" test data into something that's easily digestible for project members. I'm assuming this would end up costing more and not performing as well as using an experienced CI provider. The advantage is that it would be more decentralized.