Feature Request: send entire certificate chain (including root ca)

[hanjo]

Hi,

I noticed that caddy will present the intermediate ca certificate and the host certificate when accessing it. This can be checked with openssl, for example:

echo "" | openssl s_client -showcerts -servername example.com -connect example.com:443

Here is an example with an internal pki:

[...]
---
Certificate chain
 0 s:
   i:CN = Caddy - ECC Intermediate
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Jan  8 06:37:14 2025 GMT; NotAfter: Jan  8 18:37:14 2025 GMT
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
 1 s:CN = Caddy - ECC Intermediate
   i:CN = Caddy - 2025 ECC Root
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Jan  6 22:07:41 2025 GMT; NotAfter: Jan 13 22:07:41 2025 GMT
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
---
[...]

In this example "1" is the intermediate ca certificate and "0" is the host certificate.

This is usually working fine, as long as the client (i.e. browser) knows and trusts the root ca certificate. However, if the certificate chain is more complex (i.e. longer), this can cause trouble, because the browser wouldn't be able to fill any gaps in the certificate chain presented by caddy.

To fix this, caddy would need to present the entire certificate chain - including the root ca certificate - to the client. I checked the documentation, but I could not find an option to enable this.

So far, I've been under the impression that it is good practice to include the entire certificate chain, which is also done by many (though not all) popular public websites, for example GitHub:

$ echo "" | openssl s_client -showcerts -servername github.com -connect github.com:443
CONNECTED(00000003)
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust ECC Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo ECC Domain Validation Secure Server CA
verify return:1
depth=0 CN = github.com
verify return:1
---
Certificate chain
 0 s:CN = github.com
   i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo ECC Domain Validation Secure Server CA
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Mar  7 00:00:00 2024 GMT; NotAfter: Mar  7 23:59:59 2025 GMT
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
 1 s:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo ECC Domain Validation Secure Server CA
   i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust ECC Certification Authority
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
   v:NotBefore: Nov  2 00:00:00 2018 GMT; NotAfter: Dec 31 23:59:59 2030 GMT
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
 2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust ECC Certification Authority
   i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
   a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA384
   v:NotBefore: Mar 12 00:00:00 2019 GMT; NotAfter: Dec 31 23:59:59 2028 GMT
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
---
[...]

As you can see, "2" is the root ca certificate, "1" is the intermediate ca certificate and "0" is the host certificate.

Therefore, I'd like to request a configuration switch to enable sending the root ca certificate in the certificate chain.

Let me know what you think.
Thanks!

[hslatman]

Hey @hanjo,

It generally is good practice to send the full chain, excluding the roots.

Roots are to be installed/enabled on relying parties (e.g. client systems such as a browser), so that there's control over which roots are trusted. If a relying party were to blindly trust a root presented by a server (without additional verification) it would be equivalent to trusting any root, and thus similar guarantees as no TLS at all. So, as a web server, it's not logical to return a root certificate as part of the "full chain".

The github.com certificate is a bit of a special case in which there are two different trust paths leading to a trusted root. The first chains to USERTrust ECC Certification Authority. The second chains to AAA Certificate Services. The second includes the first chain in full: an example of cross-signing. This is documented in Sectigo's knowledge base: https://support.sectigo.com/articles/Knowledge/Sectigo-Chain-Hierarchy-and-Intermediate-Roots. This is done for compatibility reasons.

[hanjo]

Hi @hslatman,

that's an interesting perspective and I can understand the logic behind, although I don't really see the risk, as no browser/client is blindly trusting a root certificate which is not in their truststore and adding it to the truststore is a manual process which requires verification by the user, regardless if the new root ca was shared by caddy or manually imported.

I'd like to elaborate my requirements some more, maybe that clarifies things:
Assume I have an existing root ca which is not directly available to caddy - say it resides on an air-gapped server for security reasons. This root CA is trusted by all the clients in my networking. I will now create an intermediate ca, have it signed by the root ca and deploy it into the caddy configuration as root ca. Caddy will now use this intermediate ca, create another short-lived intermediate ca underneath and then the host certificates.
When a client visits such a host, there will be a certificate warning, because the client is not presented with the first intermediate ca (only the second one) and thus cannot create the trust link to the original trusted root ca. In this scenario, if caddy would be sharing the entire certificate chain (from caddy's point of view that is), it would work.

Therefore, I still believe such a configuration option would be beneficial, albeit I agree that by default this should probably be disabled, like it is now.

[bt90]

I will now create an intermediate ca, have it signed by the root ca and deploy it into the caddy configuration as root ca.

Seems like an elaborate way to shoot yourself in the foot? It's an intermediate certificate, just treat it like that.

[hanjo]

It might be worth considering that there are specific scenarios where having the option to send the entire certificate chain could add value, like I tried to point out. This feature request aims to address those use cases, providing flexibility for users who need it.

[mholt]

I'm not sure I'm convinced.

This root CA is trusted by all the clients in my networking. I will now create an intermediate ca, have it signed by the root ca and deploy it into the caddy configuration as root ca. Caddy will now use this intermediate ca, create another short-lived intermediate ca underneath and then the host certificates.

So you have:

Root --> Intermediate 1 --> Intermediate 2 --> Leaf

and Root is already trusted.

I think if a client is presented with Intermediate 2 and Leaf, it should be trusted by Root, unless the chain was formed incorrectly, no? I could be wrong though.

[hanjo]

Thanks @mholt for entertaining this idea.

So you have:

Root --> Intermediate 1 --> Intermediate 2 --> Leaf

and Root is already trusted.

This is exactly the scenario. The client trusts "Root", so in turn any certificate where "Root" is the issuer is also trusted. When the client requests the site through caddy, caddy will present the two certificates "Leaf" and "Intermediate 2". "Leaf" is issued by "Intermediate 2", so not trusted (yet). "Intermediate 2" is issued by "Intermediate 1", so not trusted (yet). "Intermediate 1" is unknown to the client (because it is neither in the trust store, nor has it been sent by caddy) and thus the client does not know by whom it's been issued. Therefore, also "Intermediate 2" and "Leaf" remain untrusted.

So there are two solutions to this issue: either I put "Intermediate 1" in the client's truststore, or the webserver (caddy in this case) needs to send also the public part of "Intermediate 1" along with the other two.

[mohammed90]

So there are two solutions to this issue: either I put "Intermediate 1" in the client's truststore, or the webserver (caddy in this case) needs to send also the public part of "Intermediate 1" along with the other two.

Why not just give Caddy the intermediate? It's already established that Caddy sends leaf and intermediate, and your issue is because you've created a disjoint in the chain by creating an additional intermediate yet serves no purpose.

https://caddyserver.com/docs/caddyfile/options#intermediate

[hslatman]

Why not just give Caddy the intermediate? It's already established that Caddy sends leaf and intermediate, and your issue is because you've created a disjoint in the chain by creating an additional intermediate yet serves no purpose.

It does serve a purpose, namely being compatible with an existing PKI. Not all certificate chains contain just a single intermediate; two or more is a perfectly valid configuration, albeit more complex (in terms of management). In general web servers do support such configurations, and it's indeed as @hanjo describes by also sending additional intermediates required to form a chain to one (or more) trusted root(s). The longer chain still shouldn't include roots, generally, though.

@hanjo: now that you describe it as an intermediate it indeed does make sense. It's an important distinction (vs. a root), hence my initial comment.

https://caddyserver.com/docs/caddyfile/options#intermediate

I haven't tested/tried it, but I suppose it can be made to work by letting the intermediate PEM file contain multiple intermediate certificates (this would require some plumbing of the slice of x509.Certificate). If the root can be just a cert (with the key being managed outside of Caddy, which seems to be possible), then the configuration isn't that much different from what it is today. A slight variation could be to introduce an additional setting for additional intermediates to be loaded and returned.

[Vigilans]

I haven't tested/tried it, but I suppose it can be made to work by letting the intermediate PEM file contain multiple intermediate certificates (this would require some plumbing of the slice of x509.Certificate).

I've tried it, it will raise this error provision pki: provisioning CA 'local': input contained more than a single PEM block as described in:

caddy/modules/caddypki/crypto.go

Lines 28 to 40 in 7099892

	func pemDecodeSingleCert(pemDER []byte) (*x509.Certificate, error) {
	pemBlock, remaining := pem.Decode(pemDER)
	if pemBlock == nil {
	return nil, fmt.Errorf("no PEM block found")
	}
	if len(remaining) > 0 {
	return nil, fmt.Errorf("input contained more than a single PEM block")
	}
	if pemBlock.Type != "CERTIFICATE" {
	return nil, fmt.Errorf("expected PEM block type to be CERTIFICATE, but got '%s'", pemBlock.Type)
	}
	return x509.ParseCertificate(pemBlock.Bytes)
	}

I am currently using this setup (root ca key not available to caddy) and it works fine:

  pki {
    ca local {
      root {
        cert /etc/pki/ca.crt
      }
      intermediate {
        cert /etc/pki/caddy/ca.crt
        key /etc/pki/caddy/ca.key
      }
    }
  }

But I think it should be supported to provide a full-chain intermediate certificate file. It serves as a very good solution to this issue: no new option introduced, users can opt-in by explicitly provide full-chain certificate as root.

  pki {
    ca local {
      root {
        cert /etc/pki/caddy/ca-fullchain.crt
        key /etc/pki/caddy/ca.key
      }
    }
  }

[mholt]

I agree -- would anyone want to patch Caddy to support a multi-cert PEM file there?

[hslatman]

I've started a draft PR: #7057.

• The root file is still expected to have a single certificate. If there are more, they are currently dropped without an error. This could be changed to be more strict.
• The intermediate file can contain multiple certificates. It is expected that the actual signer is the first certificate, the intermediate that signed the signer is next, etc. Currently no further verification of the chain is performed, but it could be made more strict.
• When loading a key pair from disk, the certificate and the signer are now compared to verify they have the same public key. This is a new check, but it shouldn't break existing deployments; just catching this case earlier.