Nix flake revamp #4096

jberthold · 2025-03-10T13:09:13Z

Rewriting the flake.nix to avoid stacklock2nix and use some-cabal-hashes for faster builds.

also removed non-standard dependencies tasty-test-reporter and profiteur
the cabal.project.freeze file is updated to be consistent with the nix build
dependency differences to stack LTS-22.23 were checked and are harmless (patch-level only, and all but 3 in test dependencies anyway)

Currently failing because of version constraints in `tasty-test-reporter` dependency `ansi-terminal`. The custom dependencies seem to be ignored.

jberthold · 2025-03-10T13:10:33Z

flake.nix

-              crypton-x509 = dontCheck hprev.crypton-x509;
-              data-fix = doJailbreak hprev.data-fix;
-              decision-diagrams = dontCheck hprev.decision-diagrams;
-              fgl = dontCheck hprev.fgl;
-              fgl-arbitrary = dontCheck hprev.fgl-arbitrary;
-              graphviz = dontCheck hprev.graphviz;
-              json-rpc = dontCheck hprev.json-rpc;
-              lifted-base = dontCheck hprev.lifted-base;
-              prettyprinter = dontCheck hprev.prettyprinter;
-              semialign = doJailbreak hprev.semialign;
-              smtlib-backends-process = dontCheck hprev.smtlib-backends-process;
-              tar = dontCheck hprev.tar;
-              text-short = doJailbreak hprev.text-short;
-              these = doJailbreak hprev.these;
-              ghc-prof = doJailbreak hprev.ghc-prof;


Some of these overrides are definitely required, and I am currently stuck trying to replicate them with some-cabal-hashes.nix

jberthold · 2025-03-10T13:12:37Z

flake.nix

+                # FIXME these two overrides cause an error in some-cabal-hashes.nix
+                # tasty-test-reporter = 
+                #     final.fetchFromGitHub {
+                #       owner = "goodlyrottenapple";
+                #       repo = "tasty-test-reporter";
+                #       rev = "b704130545aa3925a8487bd3e92f1dd5ce0512e2";
+                #       sha256 = "sha256-9CN59Im0BC3vSVhL85v5eXPYYoPbV3NAuv893tXpr/U=";
+                #     };
+                # profiteur =
+                #     final.fetchFromGitHub {
+                #       owner = "goodlyrottenapple";
+                #       repo = "profiteur";


These two are custom dependencies from github (forks of repositories that cause dependency issues). The code here follows the some-cabal-hashes example but is seemingly ignored. I already removed all code that uses tasty-test-reporter, we can remove the profiteur consumer, too and forget about this issue.

jberthold · 2025-03-10T13:13:26Z

flake.nix

+                # FIXME likewise, these two `dontCheck` (one inlined) are seemingly ignored.
+                hfinal.decision-diagrams = hfinal.overrideCabal hprev.decision-diagrams 
+                    (drv: { doCheck = false; }); # build runs failing tests
+                hfinal.json-rpc = final.haskell.packages.dontCheck hprev.json-rpc; # 1.0.4 marked as broken!


This dontCheck setting is somehow also ignored, I keep getting a build failure because of a failing test in decision-diagrams

This should be due to the attribute names that are being overridden, here. In this override example, you have to use the haskell package names directly, if I understand correctly. Therefore, currently, the original variants of the packages are being used instead.

Suggested change

# FIXME likewise, these two `dontCheck` (one inlined) are seemingly ignored.

hfinal.decision-diagrams = hfinal.overrideCabal hprev.decision-diagrams

(drv: { doCheck = false; }); # build runs failing tests

hfinal.json-rpc = final.haskell.packages.dontCheck hprev.json-rpc; # 1.0.4 marked as broken!

decision-diagrams = hfinal.overrideCabal hprev.decision-diagrams

(drv: { doCheck = false; });

json-rpc = hfinal.dontCheck hprev.json-rpc;

I tried this earlier, but nix build (--dry-run or not) complained about the hfinal.overrideCabal (or hfinal.dontCheck) as a missing attribute. Changing that to prev.haskell.lib.{overrideCabal,dontCheck} worked better 😏 yet still the build is failing on decision-diagrams (but differently now)

flake.nix

juliankuners · 2025-03-11T18:28:03Z

Okay, I checked out the branch and took a closer look at the build. This is the error message: "cabal2nix: user error (*** Found neither a .cabal file nor package.yaml. Exiting.)".

Nixpkgs provides only one set of haskell packages per compiler version, with exactly one version of a package each. Also, only one compiler version is usually maintained. A huge percentage of haskell packages is broken, though a broken package can relatively easily be fixed according to documentation. The set of haskell package derivations from stackage is automatically created, see here, in accordance to a stackage.yaml file.

The haskell-backend currently instead uses stacklock2nix to create a custom set of haskell packages on the spot, in this case with lts-22.23. The nixpkgs revision currently used instead provides the nixpkgs based set of packages that also instead relies on lts-22.22. While the difference is not large, it still explains why the json-rpc package is broken and marked as broken when not using stacklock2nix. The same is probably going on for the package decision-diagrams. Therefore, both packages derivations, and maybe some of the other dependencies, must be fixed. Again, according to nix documentation this should not be too hard.

Also, I took a look at the nixpkgs master and json-rpc is still marked as broken.

jberthold · 2025-03-11T22:57:44Z

Okay, I checked out the branch and took a closer look at the build. This is the error message: "cabal2nix: user error (*** Found neither a .cabal file nor package.yaml. Exiting.)".

Yes, correct. Which seems rather strange. I tried poking around in the /nix/store to inspect what it is trying to build but could not identify the directory it is working with (if the directory even exists).

The haskell-backend currently instead uses stacklock2nix to create a custom set of haskell packages on the spot, in this case with lts-22.23. The nixpkgs revision currently used instead provides the nixpkgs based set of packages that also instead relies on lts-22.22.

The configuration also uses a non-standard hashable package (unfortunately), so we would anyway be outside the canonical package set (the cabal.project.freeze file is for that purpose).

While the difference is not large, it still explains why the json-rpc package is broken and marked as broken when not using stacklock2nix. The same is probably going on for the package decision-diagrams. Therefore, both packages derivations, and maybe some of the other dependencies, must be fixed. Again, according to nix documentation this should not be too hard.

Maybe not hard but I have no idea at all how to approach the current issue with decision-diagrams...
It is good that we got it one step further because the overrides seem to work now, though. We will probably need more overrides (we had more in the old flake but I don't want to just cargo-cult them).

juliankuners · 2025-03-12T12:59:15Z

I continued working on the flake and pushed my progress to this branch. First, I tried to make the build work without some-cabal-hashes. Doing this led me to avoid the previous build error regarding the missing cabal project file. Apparently, this was caused by the way the previous flake used some-cabal-hashes. The overrides attribute of some-cabal-hashes is intended to be used solely to either override the version number of a haskell package or override the package source with a derivation, e.g. fetched via fetchFromGitHub or via flake inputs. This was why the overridden packages were broken. In particular, the overrides attribute of some-cabal-hashes can be used identical to packageSourceOverrides.

I added the z3 overlay from the original flake and the develop shells still need to be ported over to the new flake. Also, I reverted fixing rv-nix-tools to a specific revision for now. In my opinion, if not specifically required for this pull request, this should be in another pull request. The way we will want to refactor the way nixpkgs is locked in the runtime verification repository should be discussed separately.

Currently, hs-backend-booster fails to build due to the wrong version of hashable. The versions of the haskell packages can be found in this nix file. Getting haskell-backend to build successfully will require version overrides, taken from the cabal.project.freeze file. I wonder whether we should parse the freeze file from nix or port the freeze file to nix entirely, and henceforth maintain the version pins in the nix file?

I should mention that overriding versions of haskell packages during testing caused an infinite recursion several times. This was caused by overriding packages that are dependencies of cabal2nix, which in turn is used to when overriding a version. I fixed this in the flake by fixing the cabal2nix derivation to an unmodified derivation from the second nixpkgs instance that was in the flake already anyway.

Okay, I checked out the branch and took a closer look at the build. This is the error message: "cabal2nix: user error (*** Found neither a .cabal file nor package.yaml. Exiting.)".

Yes, correct. Which seems rather strange. I tried poking around in the /nix/store to inspect what it is trying to build but could not identify the directory it is working with (if the directory even exists).

Derivations are built in a temporary directory. Upon failure or success, this directory is deleted. To inspect a failing derivation build, you can pass the flag --keep-failed to nix build. Also, there are methods to debug the build process of a derivation step by step, which makes nix build debugging in some situations so much more comfortable.

The haskell-backend currently instead uses stacklock2nix to create a custom set of haskell packages on the spot, in this case with lts-22.23. The nixpkgs revision currently used instead provides the nixpkgs based set of packages that also instead relies on lts-22.22.

The configuration also uses a non-standard hashable package (unfortunately), so we would anyway be outside the canonical package set (the cabal.project.freeze file is for that purpose).

I see, fixing the versions should work now with the new commits from my branch.

While the difference is not large, it still explains why the json-rpc package is broken and marked as broken when not using stacklock2nix. The same is probably going on for the package decision-diagrams. Therefore, both packages derivations, and maybe some of the other dependencies, must be fixed. Again, according to nix documentation this should not be too hard.

Maybe not hard but I have no idea at all how to approach the current issue with decision-diagrams... It is good that we got it one step further because the overrides seem to work now, though. We will probably need more overrides (we had more in the old flake but I don't want to just cargo-cult them).

See above, this should not be broken anymore.

jberthold · 2025-03-13T02:23:06Z

Thanks for this, I like how it looks now with the several overlay functions!

Do you have push access to this repository? (if not we will fix that, you definitely should).

Will continue on this path, investigating the hashable issue and other overrides that we might need.

… dev-tools

juliankuners · 2025-03-14T12:09:33Z

Thanks for this, I like how it looks now with the several overlay functions!

Do you have push access to this repository? (if not we will fix that, you definitely should).

Will continue on this path, investigating the hashable issue and other overrides that we might need.

Thank you for giving me write access.

I have pushed a new branch that fixed the last two encountered build issues: https://github.com/runtimeverification/haskell-backend/commits/nix-flake-revamp-fixbuild/

I added the missing develop shells, overlays, and the missing prelude-kore output. Also, I adapted the packages output to match the packages output of the previous flake.

The previous flake had a default overlay output that included the output of stacklock2nix. We can not provide that output anymore, as any repository that uses the default overlay would be broken and needs to be updated. Though, I quickly glanced at a few repositories and couldn't find any that uses this default overlay! I also find it unlikely anyway.

@jberthold Could you please add a commit that updates the documentation regarding stack, cabal, and the updated build system? I already replaced the mentions of stacklock2nix and included some-cabal-hashes.

When building with cabal in a nix develop shell, cabal rejects the haskell packages provided by nix due to conflicts with the version specified in the freeze file.

juliankuners · 2025-03-14T13:49:08Z

I ran CI on the new branch and it failed: https://github.com/runtimeverification/haskell-backend/actions/runs/13856419943

I pushed two new commits to this branch that should fix the issues the CI is running into: https://github.com/runtimeverification/haskell-backend/commits/nix-flake-revamp-fixbuild/

One thing I did was remove the cabal.project.freeze file, as the cabal build was failing within a nix develop shell, as the version provided by nixpkgs and the flake mismatched the versions specified in the freeze file.

juliankuners · 2025-03-14T16:12:24Z

CI on the new branch ran successfully this time: https://github.com/runtimeverification/haskell-backend/actions/runs/13858014797

…ted option)

jberthold · 2025-03-19T08:10:27Z

Tested with kevm and kontrol performance scripts. No failures, kevm with minor slowdown (maybe from other load on the machine during the test).

README.md

jberthold added 7 commits March 7, 2025 10:12

add hpack-generated cabal files for nix build using cabal

226d431

pin rv-utils (nixpkgs supplier) to current master

c8c29b0

WIP messy attempt at rewriting the nix flake with some-cabal-hashes

82822c0

add some-cabal-hashes.nix as a file (with license)

9d0802d

WIP rebooted the flake.nix file based on some-cabal-hashes example

201c0b0

Currently failing because of version constraints in `tasty-test-reporter` dependency `ansi-terminal`. The custom dependencies seem to be ignored.

remove tasty-test-reporter dependency

9743c37

more WIP comments on non-working overrides

e0f593b

jberthold added the do not merge label Mar 10, 2025

jberthold commented Mar 10, 2025

View reviewed changes

juliankuners reviewed Mar 10, 2025

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

progress adjusting overrides, allowing broken packages in flake

dc1547b

juliankuners added 5 commits March 12, 2025 11:08

revert fixing rv-utils to specific revision

51dab45

WIP fix broken haskell dependencies without some-cabal-hashes

f64cbde

add z3 overlay

823dd41

add some-cabal-hashes

aedf7a9

add all-cabal-hashes to flake inputs instead

5c44181

jberthold added 8 commits March 13, 2025 14:41

WIP NON-WORKING attempts to override the hashable package

5a3044a

revert unsuccessful overriding attempts

71a5a5a

remove all-cabal-hashes overlay (use nixpkgs one), override hashable

037433b

add jailbreaks (_before_ some-cabal-hashes) due to old hashable version

01555e8

override tar library with a version present in nixpkgs all-cabal-hashes

e904e26

style-shell, including hpack (although cabal files checked in)

7167337

add separate executables (for CI and kup)

38f4ece

remove profiteur-dependent time-profile tool, remove LICENSE ref from…

ad71a70

… dev-tools

jberthold added 3 commits March 14, 2025 16:25

put back LICENSE into cabal file

6e1609c

add tools to cabal shell

d85c751

copy some dontCheck declarations from old flake

ff7f9cd

jberthold force-pushed the nix-flake-revamp branch from a4febc8 to ff7f9cd Compare March 14, 2025 07:29

juliankuners added 9 commits March 14, 2025 12:50

fix build

08b4ac5

formatting

9f79782

remove some-cabal-hashes file that is already part of inputs

9c012be

add haskell-language-server package

9b46902

match flake output packages from previous flake

b717a43

add develop shells

e911685

add overlays

a8c8389

add prelude-kore output

8a5efed

update documentation

fb88871

juliankuners added 2 commits March 14, 2025 14:13

remove cabal.project.freeze file

341a2c4

When building with cabal in a nix develop shell, cabal rejects the haskell packages provided by nix due to conflicts with the version specified in the freeze file.

fix overlays and other non-system flake outputs

3f3d83d

re-add freeze file, align stack and cabal build

4443fc9

jberthold removed the do not merge label Mar 19, 2025

jberthold marked this pull request as ready for review March 19, 2025 07:02

update kontrol perf-script to not call poetry with --no-update (outda…

578aed8

…ted option)

juliankuners requested changes Mar 19, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

Refresh nix section in README.md

7efd8a6

jberthold requested a review from juliankuners March 19, 2025 11:30

juliankuners approved these changes Mar 19, 2025

View reviewed changes

jberthold merged commit 82db2df into master Mar 20, 2025
6 checks passed

jberthold deleted the nix-flake-revamp branch March 20, 2025 02:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nix flake revamp #4096

Nix flake revamp #4096

jberthold commented Mar 10, 2025 •

edited

Loading

jberthold Mar 10, 2025

jberthold Mar 10, 2025

jberthold Mar 10, 2025

juliankuners Mar 10, 2025

jberthold Mar 11, 2025

juliankuners commented Mar 11, 2025

jberthold commented Mar 11, 2025

juliankuners commented Mar 12, 2025 •

edited

Loading

jberthold commented Mar 13, 2025

juliankuners commented Mar 14, 2025

juliankuners commented Mar 14, 2025

juliankuners commented Mar 14, 2025

jberthold commented Mar 19, 2025

Nix flake revamp #4096

Nix flake revamp #4096

Conversation

jberthold commented Mar 10, 2025 • edited Loading

jberthold Mar 10, 2025

Choose a reason for hiding this comment

jberthold Mar 10, 2025

Choose a reason for hiding this comment

jberthold Mar 10, 2025

Choose a reason for hiding this comment

juliankuners Mar 10, 2025

Choose a reason for hiding this comment

jberthold Mar 11, 2025

Choose a reason for hiding this comment

juliankuners commented Mar 11, 2025

jberthold commented Mar 11, 2025

juliankuners commented Mar 12, 2025 • edited Loading

jberthold commented Mar 13, 2025

juliankuners commented Mar 14, 2025

juliankuners commented Mar 14, 2025

juliankuners commented Mar 14, 2025

jberthold commented Mar 19, 2025

jberthold commented Mar 10, 2025 •

edited

Loading

juliankuners commented Mar 12, 2025 •

edited

Loading