CI GHC 9.4-alpha and edit Cabal/src/Distribution/Simple/GHC.hs to accept GHC 9.4

[Mikolaj]

Apparently cabal now yells

"Warning: Unknown/unsupported 'ghc' version detected (Cabal 3.8.0.20220526 supports 'ghc' version < 9.4)"?

when somebody uses GHC 9.4 and that's rather embarrassing and puts a pressure on us to create another point release ASAP when GHC 9.4-final is released. It would be ideal to add GHC 9.4 to CI and to Cabal/src/Distribution/Simple/GHC.hs (I can do the latter rather easiliy) before cabal's 3.8 release.

[Mikolaj]

This is blocked on haskell/actions#93 and haskell/actions#107.

In the worst case, we are going to release 3.8 without support for GHC 9.4 and, at some point, we may push a point release to support GHC 9.4. If haskell/actions doesn't support GHC 9.4, that's a good enough reason to yell "Unknown/unsupported 'ghc' version detected".

[andreasabel]

Haskell-CI supports 9.4: haskell-CI/haskell-ci#588

[fgaz]

Or we can add it without ci. A local test run should be enough, there isn't much to brwak

[Mikolaj]

Yes, it's our own decision

1. how much weight should our formal recognition of a GHC version carry
2. how alarming the "unsupported" warning should be
3. how spammy we want to be when the user uses an unsupported GHC version

My gut feeling is

1. once we recognize a GHC version, we need a single full test/bootstrap/etc run on major architectures for each new point release of the major release that started supporting the new GHC; normally CI does that for us, but this can be done manually every time on gitlab, I suppose; volunteers welcome
2. the warning is just right
3. we could be less spammy, e.g., record the offending GHC version somewhere in dist-newstyle and don't warn any more about that GHC until cabal clean

[jneira]

Haskell-CI supports 9.4: haskell-CI/haskell-ci#588

Nice!, if haskell-ci is able to generate a github workflow which uses ghc alpha/pre releases via templates, we can do the same writing a direct workflow at hand and using our validate.sh script.
@robx tried to do it with gha and then tried to replace gha with ghcup.
But maybe we could do a separate workflow if it is too different to existing ones.

[gbaz]

I ran a regression testsuite (cabal-testsuite) against preleaseses of cabal and ghc on macos both x86 and M1.

cabal-tests --with-cabal /Users/myuser/.ghcup/bin/cabal --with-ghc /Users/myuser/.ghcup/ghc/9.4.0.20220623/bin/ghc

There were a ton of unexpected fails, but eyeball inspection reveals, best I can tell, that this is because the golden output differed precisely because of the unexpected addition of the above warning (namely: "Warning: Unknown/unsupported 'ghc' version detected (Cabal 3.8.0.20220526 supports 'ghc' version < 9.4)").

To reiterate our verbal discussion -- the purpose of releasing cabal 3.8 is to support 9.4, and releasing it with this warning would be very silly, and we should fix it. This is to say that 9.4 is in our support window -- i.e. cabal 3.8 not working with 9.4 would be a bug, while cabal 3.6 not working with 9.4 would simply be "an unfortunate consequence that can be resolved by updating to the latest cabal"

[gbaz]

Oh -- further motivation for changing this -- the warning is not in cabal-install, but Cabal-the-library. So its double-nuts to have the library that literally ships with ghc 9.4 warn about not supporting it!

[gbaz]

I submitted a PR on this. With that PR, the testsuite succeeds with no fails, both M1 and x86

[Mikolaj]

Thanks a lot. How many tests have been skipped for you? I've got 13, mostly "SKIP no Cabal for GHC".

Could somebody please compile and run the tests on Windows, with the commandline @gbaz gives above (and from the @gbaz's PR; the following worked for me to get the PR on top of master branch: git merge upstream/gb/bump-ghc-known-versions-to-9.4)?

I've just run the tests on Linux, building via cabal build exe:cabal cabal-tests using GHC 9.2.3 and with the precise commandline /home/mikolaj/r/cabal/dist-newstyle/build/x86_64-linux/ghc-9.2.3/cabal-testsuite-3/build/cabal-tests/cabal-tests --with-ghc ~/.ghcup/ghc/9.4.0.20220623/bin/ghc --with-cabal /home/mikolaj/r/cabal/dist-newstyle/build/x86_64-linux/ghc-9.2.3/cabal-install-3.9.0.0/x/cabal/build/cabal/cabal. There are only two failures:

...
cabal-testsuite/PackageTests/NewBuild/T3827/cabal.test.hs                                                                         FAIL (unexpected success) (2.53s)
...
creating setup-per-component.dist/work/dist/doc/html/Includes2
Running: /home/mikolaj/.ghcup/ghc/9.4.0.20220623/bin/haddock-ghc-9.4.0.20220623 '@setup-per-component.dist/work/dist/doc/html/Includes2/haddock-response31842-1.txt'
   0% (  0 /  3) in 'Mine'
  Missing documentation for:
    Module header
    Mine (mylib/Mine.hs:3)
    mine (mylib/Mine.hs:4)
haddock: internal error: haddock:iface
CallStack (from HasCallStack):
  error, called at utils/haddock/haddock-api/src/Haddock/Interface.hs:224:16 in main:Haddock.Interface
...
SKIPPED 13 tests
UNEXPECTED OK: cabal-testsuite/PackageTests/NewBuild/T3827/cabal.test.hs
UNEXPECTED FAIL: cabal-testsuite/PackageTests/Backpack/Includes2/setup-per-component.test.hs cabal-testsuite/PackageTests/Backpack/Includes2/setup-external.test.hs cabal-testsuite/PackageTests/Backpack/Includes3/setup-external-ok.test.hs

Any ideas what may be causing these?

I think users can reasonably expect we at least ran our biggest testsuite once with GHC 9.4 on all major architectures given that we don't blacklist GHCs, but we explicitly whiltelist them and we don't whitelists ten GHCs a week. BTW, we should repeat the test once GHC includes the new 3.8 Cabal library, which would probably coincide with GHC 9.4.1 release. I will let @bgamari know 3.8 branch is ready for renewed inclusion in GHC once #8260 is merged and backported.

[gbaz]

I have 11 skips.

T3827 is known dodgy in both directions: #8032

In my run of the testsuite (I used master's testsuite not the branch's) the unexpected fail on Includes2 was instead marked as "OK (known failure)".

[Mikolaj]

Yes, I used the tests from master branch as well; "on top of master branch". Anyway, thank you for the explanation, I think we are fine.

Windows anybody?

[jneira]

will try to run the test suite asap in Windows, but ideally we should run it in ci.
I think it is possible, as hackage-ci has demonstrated but it would require some effort (once we got it we could reuse to test any ghc pretelease with small changes)

[Mikolaj]

Since @jneira is currently busy, could somebody else run our test suites from branch 3.8 on Windows? If that's done, I'd gladly merge and backport this PR and 3.8 release would not have any major blocker any more.

[ulysses4ever]

@Mikolaj I got a bunch of unexpected fails on a fresh Windows box. Not sure if it's my environment to blame or what. Maybe having a comparison point would make sense. E.g. 3.6? Here's the list:

UNEXPECTED FAIL: cabal-testsuite\PackageTests\TestSuiteTests\ExeV10\setup-no-tix.test.hs cabal-testsuite\PackageTests\Regression\T5782Diamond\cabal.test.hs cabal-testsuite\PackageTests\Regression\T5386\cabal.test.hs cabal-testsuite\PackageTests\Regression\T4986\cabal.test.hs cabal-testsuite\PackageTests\Regression\T4202\cabal.test.hs cabal-testsuite\PackageTests\Regression\T3294\setup.test.hs cabal-testsuite\PackageTests\NewSdist\MultiTarget\valid-and-test-suite.test.hs cabal-testsuite\PackageTests\NewSdist\MultiTarget\target-remote-package.test.hs cabal-testsuite\PackageTests\NewSdist\MultiTarget\multi-target.test.hs cabal-testsuite\PackageTests\NewSdist\MultiTarget\multi-list-sources.test.hs cabal-testsuite\PackageTests\NewSdist\MultiTarget\multi-archive-to-stdout.test.hs cabal-testsuite\PackageTests\NewSdist\MultiTarget\list-sources-output-dir.test.hs cabal-testsuite\PackageTests\NewSdist\MultiTarget\all.test.hs cabal-testsuite\PackageTests\NewSdist\MultiTarget\all-test-sute.test.hs cabal-testsuite\PackageTests\NewSdist\MultiTarget\all-output-dir.test.hs cabal-testsuite\PackageTests\NewSdist\ManyDataFiles\many-data-files.test.hs cabal-testsuite\PackageTests\NewSdist\Globbing\cabal.test.hs cabal-testsuite\PackageTests\NewFreeze\FreezeFile\new_freeze.test.hs cabal-testsuite\PackageTests\NewFreeze\Flags\new_freeze.test.hs cabal-testsuite\PackageTests\NewFreeze\BuildTools\new_freeze.test.hs cabal-testsuite\PackageTests\NewConfigure\LocalConfigOverwrite\cabal.test.hs cabal-testsuite\PackageTests\NewConfigure\ConfigFile\cabal.test.hs cabal-testsuite\PackageTests\NewBuild\MonitorCabalFiles\cabal.test.hs cabal-testsuite\PackageTests\Init\init.test.hs cabal-testsuite\PackageTests\Init\init-legacy.test.hs cabal-testsuite\PackageTests\Init\init-interactive.test.hs cabal-testsuite\PackageTests\Init\init-interactive-legacy.test.hs cabal-testsuite\PackageTests\Init\init-interactive-ghc2021.test.hs cabal-testsuite\PackageTests\Init\init-backup.test.hs cabal-testsuite\PackageTests\CCompilerOverride\setup.test.hs

[gbaz]

The right comparison would be the same cabal version but vs. ghc 9.2 instead of the 9.4 prerelease?

[ulysses4ever]

Oh, I think I used a wrong GHC (8.10), thanks for pointing to it! I'll try 9.4 and if errors persist 9.2. This will take some time (the box is slow).

[ulysses4ever]

I'm currently stuck on https://gitlab.haskell.org/haskell/ghcup-hs/-/issues/386, so ETA for finishing this is unknown.

[jneira]

I am trying to run the validate WF with ghc-9.4 here: #8264
But only for Linux and Mac OS for now

[jneira]

@ulysses4ever just in case, last time I ran it the test suite only worked in a msys2 bash console