[bug]Cannot use repositories with a custom CA and one with a system one at the same time

[numo68]

Describe the bug

Debian bookworm, Conan 2.20.1

Hello,

not sure whether this is a bug or a feature request. I need to access both the conancenter and the repository on the organization's infrastructure. The infrastructure uses a proxy to go outside. The Linux (actually a container) is configured by copying the proxy certificate to /usr/local/share/ca-certificates/xxx-ca.crt, then running update-ca-certificates. This is enough for curl so the configuration is valid, but Conan seems to ignore it.

# conan remote list
conancenter: https://center2.conan.io [Verify SSL: True, Enabled: True]
xxx-fed: https://our.org/artifactory/api/conan/xxx-fed [Verify SSL: True, Enabled: True]
# curl https://center2.conan.io/v1/ping [OK]
# curl https://our.org/artifactory/api/conan/xxx-fed/v1/ping [Also OK]
# conan create ...
ERROR: Package 'xxx/1.0.0' not resolved: HTTPSConnectionPool(host='center2.conan.io', port=443): Max retries exceeded with url: /v1/ping (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)')))
# conan create -cc core.net.http:cacert_path=/usr/local/share/ca-certificates/xxx-ca.crt ...
ERROR: Package 'xxx/1.0.0' not resolved: HTTPSConnectionPool(host='our.org', port=443): Max retries exceeded with url: /artifactory/api/conan/xxx-fed/v1/ping (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:992)')))

I know from my own experience that CA handling is less than optimal with Python based software, so this is not that surprising. Is there any trick to use both system and user-provided CA certificates, except switching the verification for one or another remote? Perhaps specify the CA per-remote in remotes.json?

Thanks

How to reproduce it

No response

[memsharded]

Hi @numo68

Thanks for your feedback.

not sure whether this is a bug or a feature request. I need to access both the conancenter and the repository on the organization's infrastructure.

I'd like to challenge a bit this initial hypothesis, and this is an important reason why our docs contain:

• https://docs.conan.io/2/devops/using_conancenter.html
• https://docs.conan.io/2/devops/conancenter/hosting_binaries.html

As it seems your organization is deeply concerned about security, then, retrieving and using pre-compiled binaries from the internet is something that in our experience is very widely forbidden by multitude of organizations, basically all of the organizations that have some security and compliance rules in place.
In practice that means that a majority of Conan users do not use ConanCenter binaries directly, but building their own binaries from source from the conan-center-index repo.

Were you aware of these recommendations? What do you think?

Regarding the CA certificates:

I know from my own experience that CA handling is less than optimal with Python based software, so this is not that surprising. Is there any trick to use both system and user-provided CA certificates, except switching the verification for one or another remote? Perhaps specify the CA per-remote in remotes.json?

Exactly, Conan implementation is just passing the conf to the underlying requests library as kwargs in the http requests:

class ConanRequester:

    def __init__(self, config, cache_folder=None):
        ...
        self._cacert_path = config.get("core.net.http:cacert_path", check_type=str)
        self._client_certificates = config.get("core.net.http:client_cert")
       
    def _add_kwargs(self, url, kwargs):
        ...
        if kwargs.get("verify", None) is not False:  # False means de-activate
            if self._cacert_path is not None:
                kwargs["verify"] = self._cacert_path
        kwargs["cert"] = self._client_certificates
        return kwargs

    def _call_method(self, method, url, **kwargs):
        try:
            all_kwargs = self._add_kwargs(url, kwargs)
            tmp = getattr(self._http_requester, method)(url, **all_kwargs)
            return tmp

So basically, they are forwarded as verify and cert arguments, as described in https://requests.readthedocs.io/en/latest/user/advanced/

But it seems you already tried it without success. Maybe it would be useful to try with a raw python-requests call, to see if there is something else that Conan is missing or doing incorrectly, and could be fixed on our side.

[numo68]

retrieving and using pre-compiled binaries from the internet is something that in our experience is very widely forbidden by multitude of organizations

Lol yes, I expected this reaction :) Yes, I am aware of those. Right now I am mostly learning and experimenting. At the end there will of course be internally hosted repository, with curated, validated and self-built versions of packages with the origin on the public internet; no way can the CI/CD hit the public internet. Still, developers will want to experiment locally, and if there is say a new version of some library out, simply download the recipe and build locally before going the circle through someone who has the responsibility and right to update that fork. And in that case it is better to at least use what is available, than just switch the verification off.

So basically, they are forwarded as verify and cert arguments, as described in
https://requests.readthedocs.io/en/latest/user/advanced/

Oh. Right, the verify for requests can be a string and this works:

{                                                                                  
 "remotes": [                                                                      
  {                                                                                
   "name": "conancenter",                                                          
   "url": "https://center2.conan.io",                                              
   "verify_ssl": "/usr/local/share/ca-certificates/xxx-ca.crt"                 
  },                                                                               
  {                                                                                
   "name": "xxx-fed",                                                     
   "url": "https://our.org/artifactory/api/conan/xxx-fed",
   "verify_ssl": true                                                              
  }                                                                                
 ]                                                                                 
}

So I'd suggest allowing to add the CA path in conan remote add / update , e.g. with --ca-path=... instead of just --insecure.

Thanks

[memsharded]

So I'd suggest allowing to add the CA path in conan remote add / update , e.g. with --ca-path=... instead of just --insecure.

Oh! It seems that you are using a loophole 😅
The arguments were booleans because we never intended/designed/considered passing a different certificate path per remote.

Now that you have reported that this can be a "feature", allowing different certificates per remotes, I think it might make sense to explicitly add it to the test suite, and document it to allow it.

I'll bring it to the team, to discuss the idea, thanks!