K8s Service Discovery Endpoint connection against https endpoints does not seem to work

[derbach]

Please, fill the following sections to help us fix the issue

What happened:
We added the HealthChecks Lib to our C# application and also created the mandatory service account, clusterrole and clusterrolebinding. The discovery in itself works. The services with Label "Healthchecks: enabled" are found and correctly resolved. However, when sending the http request, there is no response (according to the log atleast). When we exec into our pod and manually issue a request via curl/wget using the Service IP (which is correctly resolved and logged from the HealthCheck Lib), we get the expected health report. Please note, that the requests work against http endpoints and only seem to fail against https.

What you expected to happen:
We would expect the requests to our service endpoint to correctly return the health status or atleast to display an error message.
It seems like the request is timing out maybe?

How to reproduce it (as minimally and precisely as possible):
Add HealthChecksUI with enabled K8sDiscovery to C# application and create mandatory k8s ressources to enable service discovery.

Source code sample:
Appsettings.json
"HealthChecksUI": { "EvaluationTimeInSeconds": 15, "HealthChecks": [ { "Name": "HTTP-Api-Basic", "Uri": "https://172.20.147.160/healthz" // this manual check works.. this exact uri is also logged by k8sdiscovery, but no connection. } ], "KubernetesDiscoveryService": { "Enabled": true, "HealthPath": "health", "ServicesLabel": "HealthChecks", "ServicesPathAnnotation": "HealthChecksPath", "ServicesPortAnnotation": "HealthChecksPort", "ServicesSchemeAnnotation": "HealthChecksScheme", "RefreshTimeOnSeconds": 10 } }

In our C# application:
app.UseRouting().UseEndpoints(config => config.MapHealthChecksUI());

services.AddHealthChecksUI(setup => { setup.UseApiEndpointHttpMessageHandler(sp => { return new HttpClientHandler { ClientCertificateOptions = ClientCertificateOption.Manual, ServerCertificateCustomValidationCallback = (httpRequestMessage, cert, cetChain, policyErrors) => true, // NOSONAR }; }); }).AddInMemoryStorage();

Anything else we need to know?:
These are our log messages:

• Start processing HTTP request GET https://172.20.147.160/healthz
  Sending HTTP request GET https://172.20.147.160/healthz
• fail: HealthChecks.UI.Core.Discovery.K8S.KubernetesDiscoveryHostedService[0]
• Error discovering service hc-test. It might not be visible

Environment:

• Development
• .NET Core version: 8.0
• Healthchecks version: 8.0.1