X509-SVID rotation not effective for both Client and Server workloads

[skeiffer]

X509-SVID certificate rotation is not taking effect for both client and server workloads.

To show this, set your workload registration to a low TTL, IE 120 seconds and then run your test workload. Once you are past the certificate expiration you will start to see issues. Depending on how your client and server are implemented you may need to restart the client or server process for the effect to show as an existing session may keep the connection alive and a new handshake make not be needed.

note, I only tested with .net10.

Client

The following error messages are displayed when the client certificate expires

Errors

Linux

Error occurred
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The decryption operation failed, see inner exception.
---> Interop+OpenSsl+SslException: Decrypt failed with OpenSSL error - SSL_ERROR_SSL.
---> Interop+Crypto+OpenSslCryptographicException: error:0A000418:SSL routines::tlsv1 alert unknown ca

Windows

Error occurred
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The decryption operation failed, see inner exception.
---> System.ComponentModel.Win32Exception (0x80090325): The certificate chain was issued by an authority that is not trusted.

Cause

When creating a new httpclient, the GetMtlsClientOptions function is used to set SslClientAuthenticationOptions. Inside the ClientCertificateContext value is being set.

csharp-spiffe/src/Spiffe/Ssl/SpiffeSslConfig.cs

Lines 57 to 65 in e2dbda2

	public static SslClientAuthenticationOptions GetMtlsClientOptions(IX509Source x509Source, IAuthorizer authorizer)
	{
	return new SslClientAuthenticationOptions
	{
	RemoteCertificateValidationCallback = (_, cert, chain, _) =>
	ValidateRemoteCertificate(cert, chain, x509Source, authorizer),
	ClientCertificateContext = CreateContext(x509Source),
	};
	}

The problem with using ClientCerficateContext is that the context will live for the life of the httpclient object and will not get refreshed, even if the certificate is correctly getting roated in the X509source object.

Fix

Use LocalCertificateSelectionCallback instead which will be called each time a connection occurs.

example:

LocalCertificateSelectionCallback = (_, _, _, _, _) => x509Source.GetX509Svid().Certificates[0],

downside to this is the intermediate CAs will not be sent to the server from the client. I am not sure if that is an issue or not with SPIFFE standard.

Server

For server, the lack of rotation results in the following error on the client:

Error occurred
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.

Cause

Not sure if this is a .net10 only issue, but there is some sort of synchronous context deadlocking issue happening when using Krestrel or maybe any program using the Microsoft.Extensions.Hosting pattern. Once the application is run via Run() the background async task for the Source object, ie X509Source will stop executing and be in a frozen state. This results in the certificate being downloaded initially and then the task freezes before any rotations can occur.

Fix

Set the RunContinuationsAsynchronously option when creating the TaskCompletionSource inside of the Source class.

example:

_initialized = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);

[vurhanau]

downside to this is the intermediate CAs will not be sent to the server from the client. I am not sure if that is an issue or not with SPIFFE standard.

It is an issue.

I was using LocalCertificateSelectionCallback. However, our setup had an intermediate CA and it was not possible to validate X509 SVIDs in this case.

In SPIFFE standards it's mentioned that X509SVID may contain certificate chain (link):

// Required. ASN.1 DER encoded certificate chain. MAY include
// intermediates, the leaf certificate (or SVID itself) MUST come first.
bytes x509_svid = 2;

Ideally, there should be a change on .NET side:

1. LocalCertificateSelectionCallback to support certificate chain
2. SslClientAuthenticationOptions to reload when underlying context is changed

Would it be possible to mitigate this issue by setting connection lifetime shorter than SVID expiration time?

[skeiffer]

Yea an upstream setting to always send the full chain in SslClientAuthenticationOptions for example would be preferable. Though I do not know the chance of that happening if requested.

I am not sure that setting the connection lifetime will help as the connection lifetime is another setting of the SocketsHttpHandler object. I don't see how the context will get refreshed.

May have to implement a custom HttpMessageHandler that creates a new SocketsHttpHandler when the context needs to be updated. This could also help when HttpClientFactory used as well, I think.