Why "unable to get local issuer certificate" when built with LibreSSL?

[EliahKagan]

As mention in #1486 (comment), I can build gitoxide in the default max configuration for the x86_64-unknown-linux-musl target using the stable-x86_64-unknown-linux-musl on an Alpine Linux 3.17 system, overcoming OpenSSL-build failures by using apk add libressl-dev, which also installs libressl3.6-libcrypto, libressl3.6-libssl, and libressl3.6-libtls as dependencies, instead of having the openssl and openssl-dev packages installed.

However, with such a build cloning https:// URLs does not seems to ever actually work:

ek@Glub ~/r/gitoxide (main =|✔)> cargo run -r --bin=gix -- clone https://github.com/Byron/gitoxide.git
    Finished `release` profile [optimized] target(s) in 1.06s
     Running `target/release/gix clone 'https://github.com/Byron/gitoxide.git'`
Error: An IO error occurred when talking to the server

Caused by:
    [60] SSL peer certificate or SSH remote key was not OK (SSL certificate problem: unable to get local issuer certificate)
ek@Glub ~/r/gitoxide (main =|✔) [1]> cargo run -r --bin=gix -- --trace clone https://github.com/Byron/gitoxide.git
    Finished `release` profile [optimized] target(s) in 0.26s
     Running `target/release/gix --trace clone 'https://github.com/Byron/gitoxide.git'`
 21:26:30 tracing INFO     run [ 166ms | 2.00% / 100.00% ]
 21:26:30 tracing INFO     ┝━ open_from_paths() [ 2.03ms | 0.14% / 1.23% ]
 21:26:30 tracing INFO     │  ┝━ gix_path::git::install_config_path() [ 1.75ms | 1.06% ]
 21:26:30 tracing DEBUG    │  │  ┕━ 🐛 [debug]: invoking git for installation config path | cmd: "git" "config" "-l" "--show-origin"
 21:26:30 tracing INFO     │  ┕━ gix_odb::Store::at() [ 58.0µs | 0.04% ]
 21:26:30 tracing INFO     ┕━ remote::Connection::ref_map() [ 160ms | 0.01% / 96.77% ]
 21:26:30 tracing INFO        ┕━ remote::Connection::fetch_refs() [ 160ms | 0.02% / 96.76% ]
 21:26:30 tracing DEBUG          ┕━ gix_protocol::handshake() [ 160ms | 96.74% ] service: UploadPack | extra_parameters: []
Error: An IO error occurred when talking to the server

Caused by:
    [60] SSL peer certificate or SSH remote key was not OK (SSL certificate problem: unable to get local issuer certificate)

This is not specific to cloning it as a nested repository. Copying the executables out to another location and invoking from there has the same problem. The text shown above is copied from Alpine Linux 3.17, but the statically linked binary seems like it should work on any system, and it does run and clone ssh:// URLs without problems both on that system and on Ubuntu 22.04 LTS. But that error also happens on Ubuntu 22.04 LTS, which has OpenSSL installed. There are no apparent differences in the error message.

I am unsure if this is a bug in gitoxide, or if I have made a mistake in the available libraries and tools for the build, or if LibreSSL should not be expected to work at all or should be expected to require further configuration on any system where a binary built with it being used, or something else. What's going on here?

I'd be happy to convert this to an issue if this is due to a bug in gitoxide. I'm not sure, though, since #1242 mentions that with OpenSSL there is an external dependency. This is even though I believe it is statically linked, with ldd and file indicating this.

ek@Glub ~/r/gitoxide (main =|✔) [1]> file target/release/{gix,ein}
target/release/gix: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, not stripped
target/release/ein: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, not stripped

So I guess part of what I'm asking here is, what is the nature of the external OpenSSL dependency that is required with the max build configuration? Is this looking for a list of trusted certificate authorities or other local configuration?

[Byron]

Thanks for sharing! This issue seems familiar actually, even though I never dug deeper.

[..], and it does run and clone ssh:// URLs without problems both on that system and on Ubuntu 22.04 LTS.

ssh:// URLs always use the system ssh binary, as there is no built-in SSH transport yet.

I am unsure if this is a bug in gitoxide, or if I have made a mistake in the available libraries and tools for the build, or if LibreSSL should not be expected to work at all or should be expected to require further configuration on any system where a binary built with it being used, or something else. What's going on here?

LibreSSL is used through curl, which is configured to some extend but not to the extend that would affect where it looks for its client-side certificate store. Something I saw once is that people created a symlink to their certificate store in a place where it looks for it. My guess is that this might also be controllable by an environment variable, but it all seems to be very dependent on LibreSSL and maybe how it was configured at build time.

So I guess part of what I'm asking here is, what is the nature of the external OpenSSL dependency that is required with the max build configuration? Is this looking for a list of trusted certificate authorities or other local configuration?

I don't really know how it works, except that it's a matter of curl and maybe the curl-sys crate allows for additional configuration.
I hope that helps.

[EliahKagan]

LibreSSL is used through curl, which is configured to some extend but not to the extend that would affect where it looks for its client-side certificate store.

That's interesting, and is itself an area where I will need to understand more than I currently do.

The curl command does work on affected systems, even though a max build of gitoxide on the same system (built when curl was already installed and working, since I used curl to obtain rustup initially to set up the toolchain) using LibreSSL fails as shown above in gix clone https://....

Or is it actually using libcurl, rather than curl? The curl-sys crate is a wrapper for libcurl.

Is the difference relevant here, though, or do curl and libcurl share configuration at least with respect to certificates?

[Byron]

Apologies, that's what I meant. curl (CLI) is never used - but the curl crate which uses curl-sys under the hood, which is a binding to libcurl, which in turn probably does interesting things (I know nothing about) at build time to pick an SSL backend.

[EliahKagan]

Thanks for the clarification. I'll look into how that works, and what sources of configuration, especially for certificates, are used.