Annoying "http: TLS handshake error" in logs

[erikgb]

What

We are òbserving some mysterious errors in the Accurate pods:

2023/10/26 13:30:27 http: TLS handshake error from 10.101.10.1:53540: EOF
2023/10/26 13:30:27 http: TLS handshake error from 10.101.10.1:53546: EOF
2023/10/26 13:40:41 http: TLS handshake error from 10.101.10.1:47184: EOF
2023/10/26 13:40:41 http: TLS handshake error from 10.101.10.1:47194: EOF
2023/10/26 14:20:31 http: TLS handshake error from 10.101.10.1:41206: EOF
2023/10/26 15:20:41 http: TLS handshake error from 10.101.10.1:58124: EOF
2023/10/26 15:39:52 http: TLS handshake error from 10.101.10.1:58912: read tcp 10.101.13.130:9443->10.101.10.1:58912: read: connection reset by peer
2023/10/26 15:50:17 http: TLS handshake error from 10.101.10.1:53224: EOF
2023/10/26 16:20:22 http: TLS handshake error from 10.101.10.1:47576: EOF

Are anyone else observing something similar? The operator seems to work as it should, so this is just annoying messages in the logs. I suspect this might be related to kubernetes/kubernetes#109022, and some suspect the root issue to be golang/go#50984.

I'll suggest addressing #103 to see if that can fix this issue.

How

Describe how to address the issue.

Checklist

• Finish implementation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions

[erikgb]

It would be nice to get to the bottom of this annoying bug. I still see this spamming the logs after upgrading to the latest version of Accurate (1.3.0):

2024/02/21 17:40:46 http: TLS handshake error from 10.101.4.1:60740: EOF
2024/02/21 18:19:35 http: TLS handshake error from 10.101.4.1:35884: EOF
2024/02/21 18:46:31 http: TLS handshake error from 10.101.4.1:46870: EOF
2024/02/21 20:06:17 http: TLS handshake error from 10.101.4.1:48064: EOF
2024/02/21 21:16:51 http: TLS handshake error from 10.101.4.1:60182: EOF
2024/02/21 21:17:25 http: TLS handshake error from 10.101.4.1:37474: EOF
2024/02/21 21:51:53 http: TLS handshake error from 10.101.4.1:43914: EOF
2024/02/21 22:07:02 http: TLS handshake error from 10.101.4.1:41566: EOF
2024/02/21 22:18:24 http: TLS handshake error from 10.101.4.1:32964: EOF
2024/02/21 22:18:24 http: TLS handshake error from 10.101.4.1:32976: EOF
2024/02/21 22:18:24 http: TLS handshake error from 10.101.4.1:32978: EOF
2024/02/21 23:06:46 http: TLS handshake error from 10.101.4.1:42334: EOF
2024/02/21 23:19:32 http: TLS handshake error from 10.101.4.1:50816: EOF
2024/02/22 00:41:02 http: TLS handshake error from 10.101.4.1:55288: EOF
2024/02/22 00:50:44 http: TLS handshake error from 10.101.4.1:33014: EOF
2024/02/22 01:17:59 http: TLS handshake error from 10.101.4.1:39812: EOF
2024/02/22 01:21:29 http: TLS handshake error from 10.101.4.1:43696: EOF
2024/02/22 01:57:11 http: TLS handshake error from 10.101.4.1:36734: EOF
2024/02/22 02:01:17 http: TLS handshake error from 10.101.4.1:56558: EOF
2024/02/22 02:20:47 http: TLS handshake error from 10.101.4.1:53726: EOF
2024/02/22 02:42:26 http: TLS handshake error from 10.101.4.1:49920: EOF
2024/02/22 03:21:56 http: TLS handshake error from 10.101.4.1:58706: EOF
2024/02/22 03:42:03 http: TLS handshake error from 10.101.4.1:34064: EOF
2024/02/22 03:51:55 http: TLS handshake error from 10.101.4.1:35442: EOF
2024/02/22 03:52:26 http: TLS handshake error from 10.101.4.1:37308: EOF
2024/02/22 04:12:17 http: TLS handshake error from 10.101.4.1:52784: EOF
2024/02/22 04:21:43 http: TLS handshake error from 10.101.4.1:52986: EOF
2024/02/22 05:33:08 http: TLS handshake error from 10.101.4.1:48738: EOF
2024/02/22 05:42:24 http: TLS handshake error from 10.101.4.1:34250: EOF
2024/02/22 06:03:35 http: TLS handshake error from 10.101.4.1:33272: EOF
2024/02/22 06:22:36 http: TLS handshake error from 10.101.4.1:40154: EOF
2024/02/22 06:34:04 http: TLS handshake error from 10.101.4.1:47704: EOF
2024/02/22 06:39:35 http: TLS handshake error from 10.101.4.1:58154: EOF
2024/02/22 06:43:36 http: TLS handshake error from 10.101.4.1:60730: EOF
2024/02/22 07:13:22 http: TLS handshake error from 10.101.4.1:49454: EOF
2024/02/22 07:53:13 http: TLS handshake error from 10.101.4.1:35338: EOF
2024/02/22 08:20:07 http: TLS handshake error from 10.101.4.1:43340: EOF
2024/02/22 08:20:07 http: TLS handshake error from 10.101.4.1:43356: EOF
2024/02/22 08:59:44 http: TLS handshake error from 10.101.4.1:40694: EOF
2024/02/22 09:22:06 http: TLS handshake error from 10.101.4.1:54174: EOF

Am I the only one observing this? 🤔 @yamatcha, what about your installations of Accurate?

[ymmt2005]

@erikgb
can you tell what is running on 10.101.4.1?

[erikgb]

can you tell what is running on 10.101.4.1?

No, not at present. Waiting for our network expert to become available. But we don't see this problem in any other operator installed in our cluster. And we have quite a few....

[yamatcha]

@erikgb
I found the same logs as the one you showed me in my environment.
I'll look into it myself.

[allenhaozi]

same error
but the webhook works fine

2024/05/17 12:23:03 http: TLS handshake error from 10.244.0.1:55642: EOF
2024/05/17 12:23:13 http: TLS handshake error from 10.244.0.1:49250: EOF
2024/05/17 12:23:13 http: TLS handshake error from 10.244.0.1:49252: EOF
2024/05/17 12:23:23 http: TLS handshake error from 10.244.0.1:41248: EOF
2024/05/17 12:23:23 http: TLS handshake error from 10.244.0.1:41244: EOF
2024/05/17 12:23:33 http: TLS handshake error from 10.244.0.1:60528: EOF
2024/05/17 12:23:33 http: TLS handshake error from 10.244.0.1:60530: EOF
2024/05/17 12:23:43 http: TLS handshake error from 10.244.0.1:46594: EOF
2024/05/17 12:23:43 http: TLS handshake error from 10.244.0.1:46592: EOF
2024/05/17 12:23:53 http: TLS handshake error from 10.244.0.1:33732: EOF
2024/05/17 12:23:53 http: TLS handshake error from 10.244.0.1:33730: EOF
2024/05/17 12:24:03 http: TLS handshake error from 10.244.0.1:55890: EOF
2024/05/17 12:24:03 http: TLS handshake error from 10.244.0.1:55882: EOF
2024/05/17 12:24:13 http: TLS handshake error from 10.244.0.1:46278: EOF
2024/05/17 12:24:13 http: TLS handshake error from 10.244.0.1:46290: EOF
2024/05/17 12:24:23 http: TLS handshake error from 10.244.0.1:55612: EOF
2024/05/17 12:24:23 http: TLS handshake error from 10.244.0.1:55614: EOF
2024/05/17 12:24:33 http: TLS handshake error from 10.244.0.1:34004: EOF