Possible solutions

There are not so many solutions I can suggest, but I have one.

In the described problems, there are two sources of evil: 1)no check for some case we "suppose" we support; 2)the check shows red and was ignored.

To address these, we need to fix 1 and fix 2, changing current development culture, including:

• Development migrates into PR-first mode instead of committing to master. Master branch is protected from a PR merge if the CI checks are not green. Want to change something -> PR -> checks green ?? Review and merge !! Re-take.
• A check failure on master branch is considered as an extreme situation and we don't move forward until it is resolved.
• Do not rely on "We assume we support some rarer than usual platforms and try not to break them, but there are no real checks around" anymore. Establish a complete list of platforms and tools we Officialy Met Expectations for and add a clear CI check for every missing point of this list.

I know it may bring in some disapproval, saying that such restrictions are not fun for developers anymore, but I am sure that it is certainly not fun for developers to debug issues introduced 40 days ago and it is not fun for people to have troubles with packaging new releases and doing other wiring and it is not fun for us all to spend more time on fixing the consequences rather than spending less time on keeping our master branch healthy preventing the consequences in the first place.

See rakudo/rakudo#3700 (comment).

Once finished, it should help with these issues: Mar 15, Jun 5, and maybe Jun 6 because CI status should become more helpful.

Do not rely on "We assume we support some rarer than usual platforms and try not to break them, but there are no real checks around" anymore. Establish a complete list of platforms and tools we Officialy Met Expectations for and add a clear CI check for every missing point of this list.

I agree with this, but technically it also means running Blin on all of these platforms. Did you know we support mipsel? We should definitely strive towards more platforms being tested, but it's probably not possible to achieve the perfection here.

We should definitely strive towards more platforms being tested, but it's probably not possible to achieve the perfection here.

Establish a complete list of platforms and tools we Officialy Met Expectations for and add a clear CI check for every missing point of this list.

It's relatively easy with RakuDist. It has pluggable design in mind where spining up a new docker image and adding it to the system is not a big deal. So far it does tests for:

• alpine
• debian
• centos
• ubuntu

We could add more images to the list ( including old CentOS and any exotic that docker support, Sparrow is super flexible as well do deal with such a variety )

The question is what kind of test do we want to ensure another commit to MoarWM/rakudo does not break stuff. If one tells me what kind of test we need here I could start implementing this on RakuDist side.

Current test workflow is that:

• download Rakudo from whateverable
• run zef install for a certain module

But yes, we can support more testing scenarios, including building Rakudo/MoarWM from source, whatever ...

Further thoughts. I am thinking about pottage.raku.org - a service for Rakudo/MoarVM smoke testing for various Linuxes, so that for every commit we:

• build moarvm
• build rakudo
• run lightweight tests to ensure OS compatibly

Every test should run for every OS/distro in the list and should not take more say 5-10 minutes, so we could catch architecture/platform dependent bugs as soon as possible and report any problems as quick as possible.

I have all the components in place (UI/Job runner - Sparky/Backend and CM tool - docker + Sparrow ) - similar to RakuDist, so there is good base to start with ...

Further thoughts. I am thinking about pottage.raku.org - a service for Rakudo/MoarVM smoke testing for various Linuxes, so that for every commit we:

* build moarvm

* build rakudo

* run lightweight tests  to ensure OS compatibly

Isn't it exactly what we do with current CI setups?

@AlexDaniel I don't know, maybe. The idea to do it for as many OS/distros/Envs as possible. Is that the case now?

AFAIK the overhaul of the build pipelines using Azure CI by @patrickbkr die cover the build of moarvm and rakudo and some testing.

I would like to extend it to automate the star release in the future too.

We don't need more CI tests. We don't need more target platforms.

What we need is reliable CI tests. They get ignored because most of the time we look at those results is wasted because it was yet another false positive. We've had Travis reporting to the IRC channel and we'd jump whenever it reported failure. But most of them were because of Travis not being able to clone a repository or other benign issues. So someone wrote an IRC bot to tell us whether a failure was a false positive or not but that vanished, too.

What we also need is more cooperation on existing CI infrastructure. We've had Travis for Linux and OS X and AppVeyor for Windows. Someone was dissatisfied because of some issues and we got CircleCI. So we've had Travis, AppVeyor and CirecleCI all with their issues. I've added https://build.opensuse.org/project/show/home:niner9:rakudo-git because I wanted coverage for issues that would appear only in packaged versions and also coverage of important modules. This regularily reports failures like t/nqp/111-spawnprocasync.t (Wstat: 6 Tests: 4 Failed: 0) and got ignored completely and instead we got this Azure Pipelines thing.

Really, please stop adding additional systems.

Instead, just make reports worth getting looked at then make it so we don't have to check 5 different websites with different user interfaces to get at the results and then start looking at the reports, point at broken commits and create reduced test cases.

@lizmat

So far, I have seen waay more false positives from CI than I have seen false negatives. It's the false positives (when CI says there's something wrong, and it's the CI that is wrong)

Then we have to eliminate false positives instead of ignoring the checks. I mean, false positives did happen, but somehow other communities do use CI, likely preventing problems with releases and broken master for weeks as we have. There are flappers in roast but nobody pushes the code without running it saying "There are flappers, so I won't" (or so I hope). :)

@AlexDaniel

I agree with this, but technically it also means running Blin on all of these platforms

Still better than suddenly breaking someone's code in the wild because we don't check. Working on a language as we do is full of Technical Difficulties anyway, this is one of them, I guess.

Did you know we support mipsel?

I had no idea and this is precisely the problem. Tomorrow someone's code on mipsel will break next release and we will be "Hmm, well, it is not stated anywhere, but I guess we kind of support that, let's patch and release again". I am not talking about perfection and I am sure suggested scheme won't eliminate point releases. However, I don't see how catching issues earlier can be seen as something wrong.

Even a wiki page stating explicitly "We support this, that, this and that" will help tremendously.

@rba

AFAIK the overhaul of the build pipelines using Azure CI by @patrickbkr die cover the build of moarvm and rakudo and some testing

Yes, this is an awesome piece of work, because we got testing for JVM and eventually relocability too. It's just that older gcc were not on the plate, which is hopefully fix-able. Then we can have one system to rule them all and given rakudo is reliably enough to not torture us with races, we would have a great tool in our toolbox.

@niner

Really, please stop adding additional systems.

We won't (I hope). More so, migration to Azure eliminated usages of Travis and AppVeyor, quite successfully, making the things easier.

Instead, just make reports worth getting looked at then make it so we don't have to check 5 different websites with different user interfaces to get at the results and then start looking at the reports, point at broken commits and create reduced test cases.

Yes. The intention here is to 1)make CI worth being respected; 2)make people see merits of it..

Saying "Current CI is bad, so one wouldn't want to use it" is odd compared to "Current CI is bad, so we should improve it and use it".

I agree with this, but technically it also means running Blin on all of these platforms
do we run Blin for ALL modules in eco system? So it takes hours for one run, does it?

My idea is to have lightweight smoke tests run ( with average run time no more 5-10 minutes ) for all supported platforms per every commit.
Is it a case now for any of the mentioned CI (Azure Devops/Circle/Travis )?

TL;DR:
1)I don't want to cover everything in the world, add platforms, add tools. I want us to clarify what we support and what not currently and make our current tools to check if the release is worthy using this checklist.
2)I don't want to spend precious time of developers more than required, so our CI should be healthy and it should be the means for avoiding breakage of master (which is not so uncommon right now as was shown in the examples).

Flappers are acceptable red flags for me. Stupid things like connectivity issues breaking builds, are not. :-) And I've looked at way too many of those.

@Altai-man I understand all that, and I agree with all you've said.
But I still need some clarification here (from you or from others), for example you said "a critical bug which lead to build failures when linking against musl, glibc alternative notably used by Alpine distro popular for CI containers"

So, do we have a test where check a source code compatibility with Alpine? So on so forth ( you can think of some other examples, say some CentOS distros that we claim to support ).

I've found a set of OS supported in Azure Pipelienes CI for Rakudo build

https://github.com/rakudo/rakudo/blob/master/azure-pipelines.yml#L41-L97

Don't see an Alpine/CentOS/Debian here

The same for moarvm - https://github.com/MoarVM/MoarVM/blob/master/azure-pipelines.yml

cc @patrickbkr

I am not picking holes 😄 , it probably works fine for the purpose of testing moar backend / rakudo in general. But probably does not cover some OS dependent issues mentioned here ...

To start the discussion of what platforms we want to have automated tests for, I have put together a list. This is open for discussion.

• x86_64 Windows 10 (actually Microsoft Windows Server 2019 Datacenter, link)
  • Java 11
  • Visual Studio 2019
• x86_64 Ubuntu 18.04 (link)
  • GCC 9
  • Clang 9
• x86_64 MacOS 10.15 (link)
  • GCC 9
  • Java 11
• x86_64 Docker CentOS 6
  • GCC 4
• x86_64 Docker Alpine latest
  • GCC latest (currently 9)
• Some big endian system
  • The platform we actually want to support here is IBM System z. No chance we get our hands on one of those.
  • SPARCs are really cheap on ebay (e.g. Sparc T3 CPU 16 Core 1,65Ghz 32GB RAM - 235 €)
  • I'd hope we are able to get Debian on this. People are familiar with Debian in contrast to say Solaris. Debian has an unofficial SPARC64 port.
  • Getting the AzureCI runner working on one of those systems might be challenging.
• Some ARM system
  • I propose a RasPi 4 with 4GB RAM, Debian armhf 32bits - such builds will run on Raspbian
  • We could also add another RasPi with arm64.
  • RasPi is an officially supported platform for the AzureCI runner.

Open questions with the above list:

• Did I miss an environment we want to support?
• Do we really want to start setting up our own hardware for testing? Who is going to host / pay the rack space and electricity? @rba ?
• Who would pay for the hardware?
• Which test would we want to run? Currently such platformy tests (apart from OS) are only present in the MoarVM CI setup. Thus only NQP tests are run. Will this suffice to make sure our stack works on these OSes? Or do we want a full rakudo test?

MacOS 11 with ARM processor as soon as there is one available?

Also I do agree with niner and lizmat that our biggest problem with CI currently is reliability. We need to have a stable CI that people are willing to not ignore.

In that regard I'd like to focus on Azure and get rid of the others. Currently Azure isn't fully reliable (see this comment). I hope we'll manage to iron these failures out. - Soonish.

@niner From my understanding OBS is a build service and not a CI service. Did I misunderstand? Is it viable to try to use OBS as a CI?

Judging by the above list the OBS could be used as a CI and build platform for about everything except MacOS and Windows.

@niner It seems OBS doesn't really market itself as a CI. The user documentation has near to nothing on the topic of using it as such. I suspect one has to bend the system into being a CI a bit. Am I right? Things I didn't find any information about:

• Building PRs
• Reporting results back to GitHub
• Viewing testresults ordered by commit in OBS itself

There is a 2013 talk by Ralf Dannert mentioning a Jenkins integration, but information on that is just as sparse.

Edit: I am interested in looking into this more. I'd really appreciate some more information on the topic though.

@melezhik it's not out of scope, that's just how OBS works. If we create a new rakudo package for every commit, it can trigger a rebuild of all module packages (on all architectures). That's essentially what Blin does, except that OBS can do it for all supported architectures without requiring us to create our own infrastructure for it. It actually sounds a bit too good to be true, but according to @niner we are allowed to do something like this, so let's try it.

When I was looking on how to create rakudo packages, OBS was the first thing I looked at. Huge platform support, backed by a FOSS company, etc. However, I found it very complicated.

This does not mean I think OBS is a bad idea. In fact, I prefer it to Microsoft Azure. I am just stating the importance of documentation and transfer of knowledge, because I suspect that @niner++ is the only expert of the platform.

@AlexDaniel I see what you're saying and with all respect what @niner has been doing with OSB, just my thoughts:

If we create a new rakudo package for every commit, it can trigger a rebuild of all module packages (on all architectures).

you don't need this to test Moar/Rakudo, it's only makes a sense if one is going to support Raku modules for certain platform

OBS can do it for all supported architectures without requiring us to create our own infrastructure for it.

Let's be real. There is no such a tool that automatically generates all platform specific packages from META specs. Even though there is AFIK progress in that way with rhel/centos presented by @niner , we should understand that it's way too harder the we could expect now, it's even hard to to it for a certain platform, there are too many bumps on road we might be not aware of now. Again do we still need it? If we are going to maintain native packages for different Linux, then it makes a sense. However I personally don't want to build a native CentOS package for Rakudo package just to test it ... But there is somewhat in the middle approach I am currently working on discussed here one might be interested ...

There is no such a tool that automatically generates all platform specific packages from META specs

I'll submit PRs to all modules that need native dependencies. No problem.

@niner++ I love your work.

I guess almost if not all issues the discussion has started with have nothing to do with Raku modules native packages. How having those packages would help us?

@melezhik yes, you're actually right, but you have to consider the big picture. OBS can allow us massive testing of everything on all architectures. It does potentially fix some of the specific points raised in this ticket (alpine issue, stability of CI, old gcc stuff), but for others (relocatability, windows) we will have to do something extra.

Aaaand we now successfully build on s390x, i.e. IBM System z :) https://build.opensuse.org/project/show/home:niner9:rakudo-git

I'm having a hard time diving into OBS with respect to setting up a MoarVM, NQP and Rakudo CI.

@niner: As you have the deepest understanding of OBS: Can you create a write up of how a CI integration with OBS could look like for us? Some guide that gives the large picture of how such a CI should work and roughly what chunks of work need to be tackled. - An actionable plan. I imagine this would help us "OBS outsiders" a lot.

just an idea. abit another dimension in this discussion. we can spin up amazon instances ( on my free tier account this is even free ) on demand using terraform, then run tests and tear instances down. it's cheap and efficient approach. just a thought.

Sparrowdo/Sparky has recently started to support such a dynamic configurations ....

@niner Thanks for the overview! Your writeup even covers testing Raku modules!
You didn't mention GitHub integration. From a quick search I came up with a Github guide to build your own CI server integration but didn't find any information of a preexisting hook for OBS. My guess is there isn't any and we need to build our own. Do you know otherwise?

You currently use the pull paradigm (OBS regularly looks for new commits). The GitHub guide suggests using the push paradigm (GitHub pushes change notifications to a CI backend). But I suspect the GitHub API is flexible enough to get it to work in a pull fashion as well. Would you recommend one over the other?

I want to work on creating a CI setup based on AzureCI to cover MacOS and Windows, and OBS for Linux. I have not yet created a detailed concept. From what I understood it's impossible to directly couple OBS and GitHub. Some intermediary that watches for changes in GitHub or is notified by GitHub on changes and pushes those changes to OBS will be necessary. I'm considering using that same intermedary to trigger Azure builds. But I'll decide that later on once I have some progress attaching OBS.

But this ticket is not about fixing up our CI, but about stability with our development process in general. @Altai-man has provided a possible solution in the very first comment of this ticket. I'd like to reignite the discussion with respect to his proposal.

• What is the stance of the core committers towards the idea of a only pull requests development model? I've read some positive voices. Are there objections?
• I think "Failure on master is an extreme condition that needs to be resolved prio 1." has a very large cultural aspect. Changing that culture won't be done by simply stating so in some ticket. What can we do to support the change?
  • One point that has been brought up is stability of our CI. Especially false positives are discouraging respecting the CI.

A PR-only-workflow could be made easier by a helper bot.

We introduce a tag merge-on-green or ready-for-merge or some such. The bot will find PRs with that tag, wait for CI to finish and merge if the tests are all green or remove the tag and ping the creator if they turn out red. This way one doesn't have to touch each PR twice, just create the PR and add the tag in one step.

The bot could also be integrated into the CI and then be able to do things like retriggering the CI on command.

Prior art: Redhat seems to have a bot supported PR workflow: https://github.com/kubernetes/community/blob/master/contributors/guide/owners.md#the-code-review-process

If it matters, I like the idea of PR-based development. That's one of the reasons I added support for branches to whateverable. You can use committable to run code on any branch to show the upcoming behavior, you can use bisectable if needed, etc. Blin also supports branches so you can pretest your commits on the ecosystem to see if anything breaks.

Establish a complete list of platforms and tools we Officialy Met Expectations for and add a clear CI check for every missing point of this list.

So after a month and a half of discussions we still don't have one and people are asking about it.

@Altai-man: I think the problem with writing such a list now (instead of later), is that we in principle want to support as many platforms as possible and the limitting factor is having a good CI and build toolchain for the platforms. So it's currently mostly not a question of what we want to support or what is sensible to support, but what we technically can manage to have a CI for.

So the best we can do at the moment is listing which platforms we currently support well. This is what we have:

Configurations we test via our CI (only looking at Azure, ignoring Travis and CircleCI):

moar, non-reloc|reloc, x86-64, Windows 10,   MSVC,                dyncall
moar, non-reloc|reloc, x86-64, MacOS 10.15,  clang 10.0,          dyncall
moar, non-reloc|reloc, x86-64, Ubuntu 18.04, gcc 7.3.0|clang 6.0, dyncall|libffi, glibc-2.27

Platforms without CI, but for which releases are built:

moar, reloc,           x86-64, CentOS 6,     gcc 4.4.7,           dyncall,        glibc-2.12

I think with the above setup we can claim to support:

• MoarVM
• on x86-64
• Windows 10
  • MSVC
• MacOS
  • clang
• Linux
  • glibc >= 2.12
  • gcc >= 4 and clang

We can improve our coverage for the above systems (e.g. an older MacOS version, CentOS 6), but above is what we have now.

I am working on improving our CI infrastructure. I hope to be able to integrate OBS as a CI system. If I am successful we have the possibility of adding more platforms to the list.

Do we somehow have to give the above list our blessing?
Where could the list live? Somewhere in the Rakudo doc/ folder?

If it helps, Rakudist now runs community modules tests for a variety of Rakudo versions (whateverable) for the following platforms:

• Ubuntu
• Debian
• Centos
• Alpine

Again, it's just a matter of spinning up a new docker image into the RakuDist, to get a new OS tested ...

I have recently started a tool aiming, at least partly mitigate the issues mentioned here. If someone is interested - https://github.com/melezhik/r3tool

@melezhik Can you elaborate a bit what r3tool does, how it is to be used and how it relates to the already in use ecosystem tester blin?

Hi @patrickbkr ! R3 does not interferes with blin, rather the it is an additional tool.

• The main idea is to register fresh rakudo issues into database (test tags) and codify test cases for them as soon as possible, even before an issue is fixed. The approach could be applied to PRs as well, but right now just say for bugs.

• Bugs test scenarios are created in a simplest possible way - by just shell/bash code ( using Raku directly is also possible for more complex scenarios). 80-90 % of bugs get golfed as raku -e 'some broken code that exits with none zero', so it's easy to achieve. This would allow for people not familiar with roast, Raku rest eco system to codify their bugs.

• Thus bug tests always get mapped 1 to 1 to related Rakudo issues - https://github.com/rakudo/rakudo/issues and bug/ bug test is a first class citizen in this approach

• Now a release manager or developer could run test cases using various filters, skipping or exclusively inducing certain groups of bugs. Examples:

Release 2021_06, open issues:

tomty --only=rc_2021_06+open --color --show-failed

Release 2021_06, closed issues but skip slow tests:

tomty --only=rc_2021_06+closed --skip=slow --color --show-failed

Release 2021_05 or 2021_05, issues:

tomty --only=rc_2021_05,rc_2021_06 --color --show-failed

Test for rakudo release 2021_06 against all bugs, but skipping open,example,slow and requiring unicode support in terminal:

tomty --env=2021_06 --skip=slow,example,open,unicode --color --show-failed

This is in essence, other useful features - easy to test against whateverable rakudos, I am not going to go in all details here, checkout r3, please.

Advantages for rakudo development/release process:

• issues/bugs oriented approach, we can always say - what/how many issues are closed in this release, and can always run tests for certain sub sets of cases/issues. Grouping tests, so on.

• issues codifications as tests, mitigating testneeded tagged issues or cases if people forget / are lazy to add tests to roaster at all. codifying issues via tests should be neither difficult nor time consuming process. with r3 it takes me seconds to codify new issues arriving to rakudo/issues. from code implementation to commit. since that everyone should be able to run this test against their local environment. maximal efficiency

• ready TDD for developers - people fixing issues might use r3 for quick tests against their local branches

• more confidence before release - regression tests are covered by CI, but we want to run slow tests, run specific tests ( for some reasons bit supported by CI ), run performance tests, run critical issues tests, so on, so forth

• automatic statistic for a release - typically would have answers for such questions as - how many issues are closed in this release? how many issues are closed between this and that release? What kind of bug are closed? performance issues? Critical modifications potentially affecting a lot of uses ? and so on, so forth ...