Order of dream and caqti-driver-postgresql in libraries stanza cause build to fail or succeed on MacOS

[NilsIrl]

On MacOS, I have been trying to build a project that depends on dream and caqti-driver-postgresql but get the following error:

% dune build      
File "bin/dune", line 3, characters 7-11:
3 |  (name main)
           ^^^^
Undefined symbols for architecture arm64:
  "_EVP_MD_get_size", referenced from:
      _ocaml_ssl_digest in libssl_stubs.a(ssl_stubs.o)
  "_SSL_get1_peer_certificate", referenced from:
      _ocaml_ssl_get_certificate in libssl_stubs.a(ssl_stubs.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "caml_startup", line 1:
Error: Error during linking (exit code 1)

The problem is caused by the order of which things are being linked. Once I ran the underyling ocamlopt.opt command run by dune and using -ccopt to force -lssl to be included at the beginning and it worked. This problem does not exist on linux.

Now switching the order of dream and caqti-driver-postgresql in the dune file cause it to build without issues:

diff --git a/bin/dune b/bin/dune
index 59504b5..1391fc2 100644
--- a/bin/dune
+++ b/bin/dune
@@ -1,5 +1,5 @@
 (executable
  (public_name datebridge)
  (name main)
- (libraries datebridge dream caqti-driver-postgresql)
+ (libraries datebridge caqti-driver-postgresql dream)
  (preprocess (pps lwt_ppx)))

[aantron]

Thank you! @paurkedal Would you be able to comment on this issue? Do we need to document this, or is there a way to address it technically?

[paurkedal]

Does it compile if the caqti-driver-postgresql dependency is completely removed?

(caqti-driver-postgresql brings in libssl though the dependency of the libpq at the C level. The obvious but likely wrong explanation would be that something earlier in the library list depends on libssl but does not have the correct C options.)

[NilsIrl]

When caqti-driver-postgresql is removed it still compiles but then there is a runtime error: "Neither caqti-driver-postgresql nor the dynamic linker is linked into the application."

Logs:

02.11.23 16:33:58.286                       Running at http://localhost:8080
02.11.23 16:33:58.286                       Type Ctrl+C to stop
02.11.23 16:33:59.500       dream.log  INFO REQ 1 GET / 127.0.0.1:58870 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/119.0
02.11.23 16:33:59.500       dream.sql ERROR REQ 1 Dream.sql_pool: cannot create pool for 'postgres://root:hunter2@localhost/datebridge': Failed to load driver for <postgres://root:_@localhost/datebridge>: Neither caqti-driver-postgresql nor the dynamic linker is linked into the application.
02.11.23 16:33:59.500       dream.log  WARN REQ 1 Aborted by: Failure("Dream.sql_pool: cannot create pool for 'postgres://root:hunter2@localhost/datebridge': Failed to load driver for <postgres://nils:_@localhost/datebridge>: Neither caqti-driver-postgresql nor the dynamic linker is linked into the application.")
02.11.23 16:33:59.500       dream.log  WARN Raised at Stdlib.failwith in file "stdlib.ml", line 29, characters 17-33
02.11.23 16:33:59.500       dream.log  WARN Called from Lwt.Sequence_associated_storage.with_value in file "src/core/lwt.ml", line 804, characters 19-23
02.11.23 16:33:59.500       dream.log  WARN Re-raised at Lwt.Sequence_associated_storage.with_value in file "src/core/lwt.ml", line 809, characters 6-15
02.11.23 16:33:59.500       dream.log  WARN Called from Lwt.Sequential_composition.try_bind in file "src/core/lwt.ml", line 2139, characters 10-14
02.11.23 16:33:59.500      dream.http ERROR REQ 1 Failure("Dream.sql_pool: cannot create pool for 'postgres://root:hunter2@localhost/datebridge': Failed to load driver for <postgres://root:_@localhost/datebridge>: Neither caqti-driver-postgresql nor the dynamic linker is linked into the application.")
02.11.23 16:33:59.500      dream.http ERROR REQ 1 Raised at Stdlib__Map.Make.find in file "map.ml", line 141, characters 10-25
02.11.23 16:33:59.500      dream.http ERROR REQ 1 Called from Logs.Tag.find in file "src/logs.ml", line 154, characters 14-32

You can reproduce the issue by running opam switch create . --deps-only && dune exec datebridge at https://gitlab.developers.cam.ac.uk/ucms/datebridge/-/commit/f8b1fe91e111406280d9011884e3fc37130ff5dd (the child commit on the trunk branch is the one that fixes the issue by re-ordering dream and caqti-driver-postgresql in bin/dune)

[paurkedal]

Yes, the runtime error is expected, but that confirms that the postgresql driver or library is involved in some way. The next I would try would be to repeat with postgresql and substituted for caqti-driver-postgresql. If that shows the same pattern, try the same with the ssl library for comparison, since it's linked directly to libssl instead of via libpq.

I don't have MacOS, but I can try to reproduce on Linux tomorrow if it's not ruled out.

[NilsIrl]

AFAIK the issue does not exist on Linux. (I initially was developing on Linux and only found out about the issue when testing to MacOS).

[paurkedal]

Ah yes, you already wrote that in the OP. Nice work with minimizing the code.

Could the problem have the same underlying cause as facebookincubator/velox#5631? That is, two packages linking against different versions of libssl?

[paurkedal]

It may also be instructive to try a further minimization, by reducing to (libraries datebridge ssl postgresql) (but is datebridge needed?) and replacing the main program with

let _digest = Ssl.digest
let _connect () = new Postgresql.connection ()

In light of the previous comment, note that the OpenSSL version used for the ssl library is decided by conf-libssl and that used for the postgresql library is decided by the libpq build.

[NilsIrl]

I reduced the program to the 2 line you suggested and replaced the libraries with the one you suggested and got the following error when building:

% dune build           
File "bin/dune", line 3, characters 7-11:
3 |  (name main)
           ^^^^
ld: Undefined symbols:
  _EVP_MD_get_size, referenced from:
      _ocaml_ssl_digest in libssl_stubs.a[2](ssl_stubs.o)
  _SSL_get1_peer_certificate, referenced from:
      _ocaml_ssl_get_certificate in libssl_stubs.a[2](ssl_stubs.o)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "caml_startup", line 1:
Error: Error during linking (exit code 1)

I then flipped the order of ssl and postgresql in bin/dune to give (libraries postgresql ssl)) and it built without issue. datebridge is not necessary and does not affect the error.

[paurkedal]

It seems almost clear now; at least the fact that some postgresql link option override those of the ssl library. As far as I can see postgresql does not add -lssl to the command line, but it does add an -L option, which points to the output of pg_config --libdir. Could it be that this directory contains an incompatible OpenSSL library?

[dsincl12]

This solved my issue although I was using cryptokit, same thing though so if anyone else stumbles on this in the future it was the ordering as well for me.

I had:

(libraries lwt cryptokit petrol caqti caqti-lwt caqti-driver-postgresql)

and got the following error on build:

File "bin/dune", line 3, characters 7-11:
3 |  (name main)
           ^^^^
ld: Undefined symbols:
  _EVP_MD_get_size, referenced from:
      _ocaml_ssl_digest in libssl_stubs.a[2](ssl_stubs.o)
  _SSL_get1_peer_certificate, referenced from:
      _ocaml_ssl_get_certificate in libssl_stubs.a[2](ssl_stubs.o)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "caml_startup", line 1:
Error: Error during linking (exit code 1)

Just changing the ordering of the libraries in the dune file to:

(libraries lwt petrol caqti caqti-lwt caqti-driver-postgresql cryptokit)

and everything built without an error. Wasted maybe 8 hours on this and went down a lot of rabbit holes with openssl and other fun stuff.

[aantron]

Thank you! What system did you observe this on? Also macOS?

[dsincl12]

yup I'm on macOS 14.2 (Sonoma)

[aantron]

@paurkedal I'd like to close this issue here in this repo, as it seems not to be caused by Dream, though it's good that it started here because Dream's users are affected by it and can search for it. Where should it be reported? Caqti? postgresql?

[paurkedal]

Caqti is also just an intermediary is this, as far as I understand the problem. Though, I'm not sure what the postgresql-ocaml library can do if, as I assumed above, the issue is that the -L option needed to locate libpq also means that a wrong version of libssl is picked up. Further, does the ssl library maybe omit an -L option since it's a standard location, but thus not overriding a non-standard location pushed to the front of the search path by postgresql? And if that is the case, maybe it can be solved with changes to linking options in either ssl or postgresql.

[aantron]

cc @toots, @anmonteiro, @mmottl about the linking options. Do you know what is causing this ordering issue on macOS?

[toots]

It would be nice if we could have the full linking command in the broken and working case to have a look at them.