opam-monorepo: [ERROR] Solving opam-provided dependencies could not find a solution

[hannesm]

So, I try with a mirage 4.3.1 to compile a unikernel https://github.com/mirage/retreat.mirage.io:

$ opam install mirage
$ ocaml --version
The OCaml toplevel, version 4.14.0
$ opam --version
2.1.2
$ mirage --version
v4.3.1
$ opam-monorepo --version
n/a
# but installed is opam-monorepo 0.3.3 via opam, no clue why it reports "n/a"
$ git clone https://github.com/mirage/retreat.mirage.io.git
$ cd retreat.mirage.io
$ git checkout f01d3c2e1b81868cd2bdb376dbfef29e00e7d183
$ mirage configure -t hvt
$ gmake lock #use make if your GNU make is make
gmake[1]: Entering directory '/usr/home/hannes/devel/mirage/retreat.mirage.io'
gmake[2]: Entering directory '/usr/home/hannes/devel/mirage/retreat.mirage.io'
using overlay repository mirage: [opam-overlays, mirage-overlays] 
[opam-overlays] no changes from git+https://github.com/dune-universe/opam-overlays.git
[NOTE] Repository opam-overlays has been added to the selections of switch 4.14.0-2 only.
       Run `opam repository add opam-overlays --all-switches|--set-default' to use it in all existing
       switches, or in newly created switches, respectively.

[mirage-overlays] no changes from git+https://github.com/dune-universe/mirage-opam-overlays.git
[NOTE] Repository mirage-overlays has been added to the selections of switch 4.14.0-2 only.
       Run `opam repository add mirage-overlays --all-switches|--set-default' to use it in all existing
       switches, or in newly created switches, respectively.

gmake[2]: Leaving directory '/usr/home/hannes/devel/mirage/retreat.mirage.io'
 ↳ generate lockfile for monorepo dependencies
==> Using 1 locally scanned package as the target.
[WARNING] Unknown variable "ocaml-system:version"
[WARNING] Unknown variable "ocaml-base-compiler:version"
[WARNING] Unknown variable "ocaml-variants:version"
[WARNING] Unknown variable "ocaml-system:version"
[WARNING] Unknown variable "ocaml-base-compiler:version"
[WARNING] Unknown variable "ocaml-variants:version"
opam-monorepo: [ERROR] Solving opam-provided dependencies could not find a solution
[WARNING] Unknown variable "ocaml-system:version"
[WARNING] Unknown variable "ocaml-base-compiler:version"
[WARNING] Unknown variable "ocaml-variants:version"
[WARNING] Unknown variable "ocaml-system:version"
[WARNING] Unknown variable "ocaml-base-compiler:version"
[WARNING] Unknown variable "ocaml-variants:version"
[WARNING] Unknown variable "ocaml-system:version"
[WARNING] Unknown variable "ocaml-base-compiler:version"
[WARNING] Unknown variable "ocaml-variants:version"
[WARNING] Unknown variable "ocaml-system:version"
[WARNING] Unknown variable "ocaml-base-compiler:version"
[WARNING] Unknown variable "ocaml-variants:version"
[WARNING] Unknown variable "ocaml-system:version"
[WARNING] Unknown variable "ocaml-base-compiler:version"
[WARNING] Unknown variable "ocaml-variants:version"
[WARNING] Unknown variable "ocaml-system:version"
[WARNING] Unknown variable "ocaml-base-compiler:version"
[WARNING] Unknown variable "ocaml-variants:version"
opam-monorepo: [ERROR] Can't find all required versions.
Selected: angstrom.0.15.0 base64.3.5.0 bigstringaf.0.9.0 bos.0.2.1+dune
          conf-which.1 emile.1.1 fpath.0.7.3+dune functoria.4.3.2
          mirage.4.3.2 ocaml-solo5.0.8.1 ocaml-src.4.14.0
          ocaml-syntax-shims.1.0.0 opam-monorepo.0.3.4 pecu.0.6 result.1.5
          solo5.0.7.5 stringext.1.6.0 uri.4.2.0
          ocaml-base-compiler&retreat-hvt ocaml-base-compiler
          ocaml-base-compiler ocaml ocaml-base-compiler
- arp -> arp.3.0.0
    User requested = 3.0.0
- astring -> astring.0.8.5+dune
    User requested = 0.8.5+dune
- base -> base.v0.15.1
    User requested = v0.15.1
- base-bigarray -> base-bigarray.base
    User requested = base
- base-bytes -> base-bytes.base+dune
    User requested = base+dune
- base-threads -> base-threads.base
    User requested = base
- base-unix -> base-unix.base
    User requested = base
- bheap -> bheap.2.0.0
    User requested = 2.0.0
- cmdliner -> cmdliner.1.1.1+dune
    User requested = 1.1.1+dune
- conf-pkg-config -> conf-pkg-config.2
    User requested = 2
- cppo -> cppo.1.6.9
    User requested = 1.6.9
- csexp -> csexp.1.5.1
    User requested = 1.5.1
- cstruct -> cstruct.6.1.1
    User requested = 6.1.1
- cstruct-lwt -> cstruct-lwt.6.1.1
    User requested = 6.1.1
- domain-name -> domain-name.0.4.0
    User requested = 0.4.0
- dune -> dune.3.6.1
    User requested = 3.6.1
- dune-build-info -> dune-build-info.3.6.1
    User requested = 3.6.1
- dune-configurator -> dune-configurator.3.6.1
    User requested = 3.6.1
- duration -> duration.0.2.1
    User requested = 0.2.1
- eqaf -> eqaf.0.9
    User requested = 0.9
- ethernet -> ethernet.3.0.0
    User requested = 3.0.0
- findlib -> findlib.1.8.1+dune
    User requested = 1.8.1+dune
- fmt -> fmt.0.9.0+dune
    User requested = 0.9.0+dune
- functoria-runtime -> functoria-runtime.4.3.2
    User requested = 4.3.2
- ipaddr -> ipaddr.5.3.1
    User requested = 5.3.1
- logs -> logs.0.7.0+dune2
    User requested = 0.7.0+dune2
- logs-syslog -> logs-syslog.0.3.1
    User requested = 0.3.1
- lru -> lru.0.3.1
    User requested = 0.3.1
- lwt -> lwt.5.6.1
    User requested = 5.6.1
- lwt-dllist -> lwt-dllist.1.0.1
    User requested = 1.0.1
- macaddr -> macaddr.5.3.1
    User requested = 5.3.1
- macaddr-cstruct -> macaddr-cstruct.5.3.1
    User requested = 5.3.1
- memtrace-mirage -> memtrace-mirage.0.2.1.2.2
    User requested = 0.2.1.2.2
- metrics -> metrics.0.4.0
    User requested = 0.4.0
- metrics-influx -> metrics-influx.0.4.0
    User requested = 0.4.0
- metrics-lwt -> metrics-lwt.0.4.0
    User requested = 0.4.0
- mirage-bootvar-solo5 -> mirage-bootvar-solo5.0.6.0
    User requested = 0.6.0
- mirage-clock -> mirage-clock.4.2.0
    User requested = 4.2.0
- mirage-clock-solo5 -> mirage-clock-solo5.4.2.0
    User requested = 4.2.0
- mirage-console -> mirage-console.5.1.0
    User requested = 5.1.0
- mirage-console-solo5 -> mirage-console-solo5.0.8.0
    User requested = 0.8.0
- mirage-crypto -> mirage-crypto.0.10.7
    User requested = 0.10.7
- mirage-crypto-rng -> mirage-crypto-rng.0.10.7
    User requested = 0.10.7
- mirage-crypto-rng-mirage -> mirage-crypto-rng-mirage.0.10.7
    User requested = 0.10.7
- mirage-flow -> mirage-flow.3.0.0
    User requested = 3.0.0
- mirage-logs -> mirage-logs.1.2.0
    User requested = 1.2.0
- mirage-monitoring -> mirage-monitoring.0.0.3
    User requested = 0.0.3
- mirage-net -> mirage-net.4.0.0
    User requested = 4.0.0
- mirage-net-solo5 -> mirage-net-solo5.0.8.0
    User requested = 0.8.0
- mirage-profile -> mirage-profile.0.9.1
    User requested = 0.9.1
- mirage-random -> mirage-random.3.0.0
    User requested = 3.0.0
- mirage-runtime -> mirage-runtime.4.3.2
    User requested = 4.3.2
- mirage-solo5 -> mirage-solo5.0.9.1
    User requested = 0.9.1
- mirage-time -> mirage-time.3.0.0
    User requested = 3.0.0
- mtime -> mtime.1.4.0+dune2
    User requested = 1.4.0+dune2
- num -> num.1.4+dune2
    User requested = 1.4+dune2
- ocaml -> ocaml.4.14.0
    User requested = 4.14.0
- ocaml-base-compiler -> ocaml-base-compiler.4.14.0
    User requested = 4.14.0
- ocaml-compiler-libs -> ocaml-compiler-libs.v0.12.4
    User requested = v0.12.4
- ocaml-config -> ocaml-config.2
    User requested = 2
- ocaml-options-vanilla -> ocaml-options-vanilla.1
    User requested = 1
- ocamlbuild -> ocamlbuild.0.14.2
    User requested = 0.14.2
- ocamlfind -> ocamlfind.1.8.1+dune
    User requested = 1.8.1+dune
- ocplib-endian -> ocplib-endian.1.2
    User requested = 1.2
- omd -> omd.2.0.0~alpha3
    User requested = 2.0.0~alpha3
- parse-argv -> parse-argv.0.2.0
    User requested = 0.2.0
- parsexp -> parsexp.v0.15.0
    User requested = v0.15.0
- ppx_cstruct -> ppx_cstruct.6.1.1
    User requested = 6.1.1
- ppx_derivers -> ppx_derivers.1.2.1
    User requested = 1.2.1
- ppxlib -> ppxlib.0.28.0
    User requested = 0.28.0
- psq -> psq.0.2.1
    User requested = 0.2.1
- ptime -> ptime.1.0.0+dune2
    User requested = 1.0.0+dune2
- randomconv -> randomconv.0.1.3
    User requested = 0.1.3
- retreat-hvt -> retreat-hvt.zdev
    User requested = zdev
- rresult -> rresult.0.7.0+dune
    User requested = 0.7.0+dune
- seq -> seq.base+dune
    User requested = base+dune
- sexplib -> sexplib.v0.15.1
    User requested = v0.15.1
- sexplib0 -> sexplib0.v0.15.1
    User requested = v0.15.1
- stdcompat -> stdcompat.19
    User requested = 19
- stdlib-shims -> stdlib-shims.0.3.0
    User requested = 0.3.0
- syslog-message -> syslog-message.1.1.0
    User requested = 1.1.0
- tcpip -> tcpip.7.1.2
    User requested = 7.1.2
- tyxml -> (problem)
    User requested = 3.0.0
    Rejected candidates:
      tyxml.3.0.0: Requires ocaml >= 3.12.1 & < 4.06.0
- uucp -> (problem)
    User requested = 0.9.0
    Rejected candidates:
      uucp.0.9.0: Requires ocaml >= 4.00.0 & < 4.06.0
- uunf -> uunf.0.9.3
    User requested = 0.9.3
- uutf -> (problem)
    User requested = 0.9.3
    emile 1.1 requires >= 1.0.0
    Rejected candidates:
      uutf.0.9.3: Incompatible with restriction: >= 1.0.0
gmake[2]: Entering directory '/usr/home/hannes/devel/mirage/retreat.mirage.io'
removing overlay repository [opam-overlays, mirage-overlays]
Repositories removed from the selections of switch 4.14.0-2. Use '--all' to forget about them altogether.
Repositories removed from the selections of switch 4.14.0-2. Use '--all' to forget about them altogether.
gmake[2]: Leaving directory '/usr/home/hannes/devel/mirage/retreat.mirage.io'
gmake[1]: *** [Makefile:36: mirage/retreat-hvt.opam.locked] Error 1
gmake[1]: Leaving directory '/usr/home/hannes/devel/mirage/retreat.mirage.io'
gmake: *** [Makefile:39: lock] Error 2

Now, due to , I get a huge chunk of output from "opam monorepo" which I have trouble to understand:

• "User requested" some versions (hint: I did not, and I am the user - see my opam file retreat-hvt.opam.txt)
• some have "(problem)", but no explanation how to solve them

If you like to reproduce with only the opam file, please take into consideration the call in the Makefile is:

OPAM is opam
UNIKERNEL_NAME is retreat-hvt
"shell ocamlc --version" is 4.14.0

The repositories in use are:
- ocaml/opam-repository at 15773f4
- dune-universe/opam-overlays at d97e352
- dune-universe/mirage-opam-overlays at 05f1c18

	@env OPAMVAR_monorepo="opam-monorepo" $(OPAM) monorepo lock --require-cross-compile --build-only $(UNIKERNEL_NAME) -l $@ --ocaml-version $(shell ocamlc --version); (ret=$$?; $(MAKE) -s repo-rm && exit $$ret)

On Dec 13th at 23:00 UTC there was a successful build with the exact same retreat.mirage.io commit (see https://builds.robur.coop/job/retreat/build/5a87812b-9a34-409c-adf9-e7c1a50e1070/ for details). The only moving part seems to be opam-repository.

Now, what can I do? After some hours, I figured "tyxml" may be related, and if I add a >= "4.5.0" constraint to tyxml (the most recent release), "opam monorepo" finds a solution. Why is that? Why is uu* not up to date in opam-overlays (is this the underlying problem?)? How can I debug why "opam monorepo" selected some versions? The "tyxml (problem) User requested = 3.0.0" makes me wonder how "opam monorepo" requests versions, and whether it can explain its selection criteria?

This is again a very painful experience to use this "opam-monorepo". I try to be patient and supportive for "opam monorepo", but every other day I encounter other trouble that makes this tool very hard to use - instead I spend hours on trying to find workarounds. I still have the feeling that nobody is using "opam monorepo" (apart from mirage 4 forces its usage), or am I really the only one running into such issues?

Would a switch to the normal opam solver fix this issue?

Apart from this issue, major issues I encounter every now and then and try to find workarounds

• duniverse loses opam patches #113
• Semantically equivalent but different "dev-repo" fields lead to build errors #118
• Inconsistent solution when incompatible packages from the same repository are selected #145
• opam monorepo fails on repo with pin-depends #262
• --ocaml-version can cause solver failures #264
• Silent failure when there is a cycle #291
• failure with a pinned opam package #326 (eventually fixed in "0.3.5")
• opam monorepo ignores upper bounds #331
• Solver can select ocaml-base-compiler and ocaml-variant at the same time #340
• What's the reasoning behind filtering packages that do not have a dev-repo field? #341 (now the "fix" is to emit a warning)
• solver does not work when latest version of a package is not compatible with requested ocaml version #342
• pull overwrites everything #348
• --ocaml-version does not support pre-releases of the compiler #351

Is it maybe worth to rethink the approach "opam monorepo" has, and maybe start writing down the exact desired semantics with the above issues in mind (the opam repository is indeed changing over time, and there are multiple opam packages in the same tarball)?

//cc @samoht @avsm

[Leonidas-from-XIV]

> $ opam-monorepo --version
n/a
# but installed is opam-monorepo 0.3.3 via opam, no clue why it reports "n/a"

I assume that usually this Cmdliner information is filled out via dune subst that we're missing. I believe this is easy enough to fix, meanwhile the information is available in opam info opam-monorepo. Thanks for the report.

"User requested" some versions (hint: I did not, and I am the user

This wording is from the opam-0install-solver when it prints what the solver issue is. I agree the wording is a bit confusing, it should be "solver has decided upon this version to satisfy the users request". Worse than the wording is that it doesn't really tell you why it has chosen this version.

After some hours, I figured "tyxml" may be related, and if I add a >= "4.5.0" constraint to tyxml (the most recent release), "opam monorepo" finds a solution. Why is that?

I feel your frustration. Looking at the output and your OPAM file I am unsure why the opam-0install-solver decided that tyxml.3.0.0 is the version to pick, especially given it doesn't even work on the selected ocaml version. opam-monorepo does (deliberately) very little handpicking because it usually doesn't have the insight on what would be best so apart from very few (mostly build packages). And the more handpicking it does the more it has to catch up with what opam-repository is doing.

In general, it's mostly about trying to solve the lines that are labelled (problem) and often times solving one problem solves the others too or finds new problems. I can't give you a good answer quickly why opam-0install-solver chose a version of tyxml that makes no sense, I can look into it but it looks like there is either a bug in the solver or in the way we add the "builds with dune" constraints.

Why is uu* not up to date in opam-overlays (is this the underlying problem?)?

It is because the packages in opam-overlays are updated manually. We had some discussions of extending Daniel's b0 to generate dune files automatically which would take a lot of busywork out of the process and could maybe even obsolete the inclusions of Daniels packages into opam-overlays but I haven't seen much progress on this lately. I started a project to do so, but even according to Daniel the b0 API is somewhat unstable and hasn't settled yet.

That said I doubt that this is the case, very rarely do packages depend on rather specific new versions of the packages (and for those that do, like fmt, uutf and cmdliner we have reasonably recent dunified releases).

Would a switch to the normal opam solver fix this issue?

A good question. The OPAM solver for sure does a better job at picking versions and explaining why constraints couldn't be fulfilled. On the other hand, the opam solver hasn't been designed to be used separately from the opam CLI so somebody more familiar with the OPAM API would need to chime in. Given its evolution I wouldn't be surprised it not a lot of the assumptions of the current OPAM workflows (all packages qualify, the repositories are specified in switches) were baked in.

Apart from this issue, major issues I encounter every now and then and try to find workarounds

Thanks for the list, I think these fall info multiple categories:

• --ocaml-version can cause solver failures #264
• --ocaml-version does not support pre-releases of the compiler #351

These are due to the mismatch of packages named ocaml in opam-repository where ocaml and ocaml-base-compiler do not have matching versions. I don't know why that is but it this keeps happening we'll need to handle --ocaml-version differently (but to do so it helps to understand why this is the case). I can try to implement a workaround but it seems a bit like a cludge and has a chance of causing other problems.

• Semantically equivalent but different "dev-repo" fields lead to build errors #118
• Inconsistent solution when incompatible packages from the same repository are selected #145
• opam monorepo ignores upper bounds #331

These are all due to it being impossible to unpack each individual tarball into the duniverse. I started tagging them as such so if we manage to implement a solution in dune (basically -p for vendored packages) then these go away. For now we have to deduplicate somehow and the way it works is via dev-repo and picking the newest tarball which kind of works, but we're both in agreement that this is a pretty ugly solution and @TheLortex' issue that multiple URLs might lead to the same projects (also if moving a repo between organizations) so even if we solve the particular .git issue it won't solve the problem. If you're aware of a better interim solution, I'd love to rework this and be able to close this.

• duniverse loses opam patches #113

Very valid concern. I can't say more than what's already written in the ticket, though, it hasn't moved since.

[hannesm]

Thanks for your elaborate answer.

It is quite concerning to read from the main maintainer

Looking at the output and your OPAM file I am unsure why the opam-0install-solver decided that tyxml.3.0.0 is the version to pick, especially given it doesn't even work on the selected ocaml version

(and this is not the first time I read something along these lines in this project)

It is because the packages in opam-overlays are updated manually

Sounds like the approach taken by "opam monorepo" does not scale very well. Last time I asked, there was some ongoing work (shell scripts or so) on at least figuring out what needs updating in "opam-overlays".

If you're aware of a better interim solution, I'd love to rework this and be able to close this.

I've still, after months of trying, still no clue about what "opam monorepo" attempts to solve. Sometimes I feel I have a handle on that aspect, but then "opam monorepo" surprises me again. Thus, please don't expect any solution from me. It sounds like all you need is figuring out in a "dune" file which library it belongs to (so, maybe a "grep public_name" is sufficient)? And then deleting all other subdirectories?

You picked some specific issues from my list and explained why you think it is as it is. But there are some more issues in the list you didn't talk about, is there any specific reason? #262 #291 #340 #342