`manager.go` "x509: certificate signed by unknown authority, unable to fully scrape metrics from source" #541

agilgur5 · 2020-06-11T22:21:05Z

I've exhausted all the options and tried A TON of different possible fixes (been at it for more than a full day) but none worked, so I am thinking this may be a bug here and writing here

What happened:

I deployed metrics-server onto a kops cluster on AWS using the Helm chart with no changes. metrics-server errors with the full logs listed below. Summarized by the title and a short snippet below:

1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-172-20-38-5.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-38-5.us-west-1.compute.internal (ip-172-20-38-5.us-west-1.compute.internal): Get https://ip-172-20-38-5.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source .....

Per the FAQ, I tried to set --kubelet-certificate-authority to the kops CA (hostPath /srv/kubernetes/ca.crt), but still got this error. I tried setting --client-ca-file to that as well and still got this error.

I tried changing --requestheader-client-ca-file (which seems to only be for kubeadm and unrelated) to this CA cert as well as to the proxy-client CA but then got the authentication.go error that that was supposed to fix per the FAQ

The only thing that worked for me was setting --kubelet-insecure-tls which made the errors disappear and made kubectl top nodes start working. But this is, of course, insecure and a MITM attack vector, but I couldn't get any other flag to work 😕
Does the --kubelet-certificate-authority flag not work? I looked at various issues and none of them really resolve it, they all say to set --kubelet-insecure-tls. Many are DNS issues which I'm not having.

What you expected to happen:

Helm installation to either work without configuration (validating against the cluster CA) or to work once --kubelet-certificate-authority is specified.

Anything else we need to know?:

I have set --authentication-token-webhook=true and --authorization-mode=Webhook as you can see below via kops config. (without these, and without performing a rolling-update after setting them, I get 401 Unauthorized instead, which is expected. Once they are set, this error happens instead).

My error and config look very similar to #362 (comment) who also asked if the --kubelet-certificate-authority flag was ignored.

I'm getting this error in multiple clusters, on multiple k8s and kops versions, all of them have the same error.

Environment:

Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): kops on AWS. v1.15 and v1.17, errors in both
Container Network Setup (flannel, calico, etc.): Cilium in some, Calico in others, errors in both
Kubernetes version (use kubectl version): v1.15.11 in some, v1.18.3 in others, errors in both
Metrics Server manifest: helm upgrade --install metrics-server with the defaults. Some changes tried as listed above
Kubelet config:

admin@ip-172-20-58-86:~$ ps -aux | grep kubelet
root      1975  3.0  1.3 649748 111596 ?       Ssl  Jun10  41:06 /usr/local/bin/kubelet --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-root=/ --client-ca-file=/srv/kubernetes/ca.crt --cloud-provider=aws --cluster-dns=100.64.0.10 --cluster-domain=cluster.local --enable-debugging-handlers=true --eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5% --feature-gates=ExperimentalCriticalPodAnnotation=true --hostname-override=ip-172-20-58-86.us-west-1.compute.internal --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --node-labels=kops.k8s.io/instancegroup=nodes,kubernetes.io/role=node,node-role.kubernetes.io/node= --non-masquerade-cidr=100.64.0.0/10 --pod-infra-container-image=k8s.gcr.io/pause-amd64:3.0 --pod-manifest-path=/etc/kubernetes/manifests --register-schedulable=true --v=2 --volume-plugin-dir=/usr/libexec/kubernetes/kubelet-plugins/volume/exec/ --cni-bin-dir=/opt/cni/bin/ --cni-conf-dir=/etc/cni/net.d/

Metrics Server logs:

Open spoiler for logs:

Most relevant snippet:

E0610 20:12:15.220770       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-172-20-38-5.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-38-5.us-west-1.compute.internal (ip-172-20-38-5.us-west-1.compute.internal): Get https://ip-172-20-38-5.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-86-41.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-86-41.us-west-1.compute.internal (ip-172-20-86-41.us-west-1.compute.internal): Get https://ip-172-20-86-41.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-36-220.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-36-220.us-west-1.compute.internal (ip-172-20-36-220.us-west-1.compute.internal): Get https://ip-172-20-36-220.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-82-244.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-82-244.us-west-1.compute.internal (ip-172-20-82-244.us-west-1.compute.internal): Get https://ip-172-20-82-244.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-70-141.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-70-141.us-west-1.compute.internal (ip-172-20-70-141.us-west-1.compute.internal): Get https://ip-172-20-70-141.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-45-12.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-45-12.us-west-1.compute.internal (ip-172-20-45-12.us-west-1.compute.internal): Get https://ip-172-20-45-12.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-52-222.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-52-222.us-west-1.compute.internal (ip-172-20-52-222.us-west-1.compute.internal): Get https://ip-172-20-52-222.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-72-191.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-72-191.us-west-1.compute.internal (ip-172-20-72-191.us-west-1.compute.internal): Get https://ip-172-20-72-191.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority]

Full logs:

$ kubectl -n kube-system logs metrics-server-8555869558-kvqdq 
I0610 20:11:15.159707       1 secure_serving.go:116] Serving securely on [::]:8443
E0610 20:11:44.938984       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-86-41.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939013       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-45-12.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939020       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-70-141.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939026       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-38-5.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939039       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-72-191.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939045       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-36-220.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939050       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-52-222.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.939056       1 reststorage.go:135] unable to fetch node metrics for node "ip-172-20-82-244.us-west-1.compute.internal": no metrics known for node
E0610 20:11:44.945326       1 reststorage.go:160] unable to fetch pod metrics for pod logging/fluentbit-cloudwatch-fluent-bit-5pfbd: no metrics known for pod
E0610 20:11:44.945347       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istiod-c7757dcf7-vz8cn: no metrics known for pod
E0610 20:11:44.945353       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-kwpbp: no metrics known for pod
E0610 20:11:44.945358       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-56676bcc8f-vpqk8: no metrics known for pod
E0610 20:11:44.945363       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/alertmanager-prometheus-operator-alertmanager-0: no metrics known for pod
E0610 20:11:44.945397       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-6j4gr: no metrics known for pod
E0610 20:11:44.945417       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-controller-manager-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945422       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-operator-5dd8f8f568-24x48: no metrics known for pod
E0610 20:11:44.945428       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-9glt8: no metrics known for pod
E0610 20:11:44.945433       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-8555869558-kvqdq: no metrics known for pod
E0610 20:11:44.945438       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube2iam-9zwtv: no metrics known for pod
E0610 20:11:44.945444       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-events-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945449       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kops-controller-2c7bp: no metrics known for pod
E0610 20:11:44.945454       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-5pjrf: no metrics known for pod
E0610 20:11:44.945459       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-apiserver-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945464       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-scheduler-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945470       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-36-220.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945475       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-scheduler-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945480       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-74z29: no metrics known for pod
E0610 20:11:44.945485       1 reststorage.go:160] unable to fetch pod metrics for pod logging/fluentbit-cloudwatch-fluent-bit-gfgzt: no metrics known for pod
E0610 20:11:44.945491       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kops-controller-v2spk: no metrics known for pod
E0610 20:11:44.945496       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-fjzm2: no metrics known for pod
E0610 20:11:44.945504       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/dns-controller-6598994b65-8fsts: no metrics known for pod
E0610 20:11:44.945510       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-56676bcc8f-vxmzh: no metrics known for pod
E0610 20:11:44.945515       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-prometheus-operator-prometheus-0: no metrics known for pod
E0610 20:11:44.945520       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-controller-manager-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945526       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-86-41.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945559       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-27wk8: no metrics known for pod
E0610 20:11:44.945566       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-apiserver-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945571       1 reststorage.go:160] unable to fetch pod metrics for pod istio-operator/istio-operator-54fc967ff8-f9jm5: no metrics known for pod
E0610 20:11:44.945576       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istiocoredns-774588b997-thb66: no metrics known for pod
E0610 20:11:44.945581       1 reststorage.go:160] unable to fetch pod metrics for pod external-secrets/external-secrets-kubernetes-external-secrets-5459f65756-7ncdf: no metrics known for pod
E0610 20:11:44.945587       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-n58ld: no metrics known for pod
E0610 20:11:44.945592       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-82-244.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945597       1 reststorage.go:160] unable to fetch pod metrics for pod kubernetes-dashboard/kubernetes-dashboard-74b4487bfc-pxfkw: no metrics known for pod
E0610 20:11:44.945602       1 reststorage.go:160] unable to fetch pod metrics for pod logging/fluentbit-cloudwatch-fluent-bit-jblrb: no metrics known for pod
E0610 20:11:44.945608       1 reststorage.go:160] unable to fetch pod metrics for pod kubernetes-dashboard/dashboard-metrics-scraper-66b49655d4-qgdj8: no metrics known for pod
E0610 20:11:44.945613       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-72-191.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945618       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-events-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945623       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube2iam-bh9hb: no metrics known for pod
E0610 20:11:44.945629       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-6nqhh: no metrics known for pod
E0610 20:11:44.945651       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945658       1 reststorage.go:160] unable to fetch pod metrics for pod logging/fluentbit-cloudwatch-fluent-bit-bkssn: no metrics known for pod
E0610 20:11:44.945663       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kops-controller-khqfh: no metrics known for pod
E0610 20:11:44.945668       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-scheduler-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945674       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-operator-987865868-vjhrv: no metrics known for pod
E0610 20:11:44.945679       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-main-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945684       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-52-222.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945689       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-main-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945720       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-jw7nb: no metrics known for pod
E0610 20:11:44.945728       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-events-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945733       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-controller-manager-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945739       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-vmjfb: no metrics known for pod
E0610 20:11:44.945744       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/grafana-5f6f8cbf75-h5kl5: no metrics known for pod
E0610 20:11:44.945749       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-apiserver-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945755       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-kube-state-metrics-5fdcd78bc-8ldlf: no metrics known for pod
E0610 20:11:44.945760       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-dn7qb: no metrics known for pod
E0610 20:11:44.945765       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/guard-76b48854ff-59df8: no metrics known for pod
E0610 20:11:44.945770       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-autoscaler-cd5745894-sz59s: no metrics known for pod
E0610 20:11:44.945775       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube2iam-ndckw: no metrics known for pod
E0610 20:11:44.945780       1 reststorage.go:160] unable to fetch pod metrics for pod logging/fluentbit-cloudwatch-fluent-bit-wmx2q: no metrics known for pod
E0610 20:11:44.945786       1 reststorage.go:160] unable to fetch pod metrics for pod external-secrets/external-secrets-kubernetes-external-secrets-5459f65756-rfh8q: no metrics known for pod
E0610 20:11:44.945790       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/cilium-dnn2k: no metrics known for pod
E0610 20:11:44.945796       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-nlmm4: no metrics known for pod
E0610 20:11:44.945801       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-70-141.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945806       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istio-ingressgateway-6cf56d9598-4xw9n: no metrics known for pod
E0610 20:11:44.945811       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube2iam-knwr6: no metrics known for pod
E0610 20:11:44.945816       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-gmwvg: no metrics known for pod
E0610 20:11:44.945821       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-ip-172-20-38-5.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945826       1 reststorage.go:160] unable to fetch pod metrics for pod monitoring/prometheus-operator-prometheus-node-exporter-5mjnv: no metrics known for pod
E0610 20:11:44.945832       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/etcd-manager-main-ip-172-20-45-12.us-west-1.compute.internal: no metrics known for pod
E0610 20:11:44.945837       1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube2iam-sprw4: no metrics known for pod
E0610 20:11:44.945842       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/prometheus-dfd976959-rkjw9: no metrics known for pod
E0610 20:11:46.238622       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istio-ingressgateway-6cf56d9598-4xw9n: no metrics known for pod
E0610 20:11:46.250946       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istiod-c7757dcf7-vz8cn: no metrics known for pod
E0610 20:12:01.255135       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istio-ingressgateway-6cf56d9598-4xw9n: no metrics known for pod
E0610 20:12:01.261228       1 reststorage.go:160] unable to fetch pod metrics for pod istio-system/istiod-c7757dcf7-vz8cn: no metrics known for pod
E0610 20:12:15.220770       1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-172-20-38-5.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-38-5.us-west-1.compute.internal (ip-172-20-38-5.us-west-1.compute.internal): Get https://ip-172-20-38-5.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-86-41.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-86-41.us-west-1.compute.internal (ip-172-20-86-41.us-west-1.compute.internal): Get https://ip-172-20-86-41.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-36-220.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-36-220.us-west-1.compute.internal (ip-172-20-36-220.us-west-1.compute.internal): Get https://ip-172-20-36-220.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-82-244.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-82-244.us-west-1.compute.internal (ip-172-20-82-244.us-west-1.compute.internal): Get https://ip-172-20-82-244.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-70-141.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-70-141.us-west-1.compute.internal (ip-172-20-70-141.us-west-1.compute.internal): Get https://ip-172-20-70-141.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-45-12.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-45-12.us-west-1.compute.internal (ip-172-20-45-12.us-west-1.compute.internal): Get https://ip-172-20-45-12.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-52-222.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-52-222.us-west-1.compute.internal (ip-172-20-52-222.us-west-1.compute.internal): Get https://ip-172-20-52-222.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-172-20-72-191.us-west-1.compute.internal: unable to fetch metrics from Kubelet ip-172-20-72-191.us-west-1.compute.internal (ip-172-20-72-191.us-west-1.compute.internal): Get https://ip-172-20-72-191.us-west-1.compute.internal:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority]

/king bug

The text was updated successfully, but these errors were encountered:

fejta-bot · 2020-09-09T22:56:32Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

agilgur5 · 2020-10-07T20:30:42Z

/remove-lifecycle stale

This issue has never been responded to...

There was a response in kubernetes/kops#6879 (comment) , but that revealed that there's a missing capability required for this to work 😕

fejta-bot · 2021-01-07T04:30:14Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2021-02-06T05:14:32Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

serathius · 2021-02-21T18:53:53Z

I think problem was resolved by kubernetes/kops#10022 on kops side and #589 on Metrics Server side.

Feel free to reopen if problem still persists.

unacceptable · 2022-04-07T23:15:27Z

For my friends on EKS make sure you have the username set (and not set to just the session name like I did):

    robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
    - rolearn: arn:aws:iam::123546789012:role/MyTeamRole
      username: {{SessionName}}
      groups:
        - system:masters
    robert ❱ kubectl top node
    error: You must be logged in to the server (Unauthorized)
    robert ❱ 1 ❱ kubectl edit configmap -n kube-system aws-auth
    configmap/aws-auth edited
    robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
        - rolearn: arn:aws:iam::123546789012:role/MyTeamRole
          username: literally_anything:{{SessionName}}
          groups:
            - system:masters
    robert ❱ kubectl top node
    NAME                                       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
    ip-10-0-3-103.us-west-2.compute.internal   341m         17%    1738Mi          52%
    ...
    robert ❱ kubectl logs -n kube-system -l app.kubernetes.io/instance=metrics-server
    E0407 22:34:45.879156       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
    E0407 22:34:49.399854       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
    E0407 22:34:50.691133       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
    E0407 22:34:51.827629       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
    E0407 22:39:07.288163       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
    E0407 22:39:08.755492       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
    E0407 22:39:09.801957       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
    E0407 22:40:32.405458       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
    E0407 22:43:09.791769       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
    E0407 22:44:14.244221       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
    robert ❱

This was referenced Jun 11, 2020

[METRICS-SERVER] New kubelet-certificate-authority flag available (v0.3.2) kubernetes/kops#6879

Closed

Metrics-server: How to configure TLS authentication instead of using --kubelet-insecure-tls? kubernetes/kops#7083

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 9, 2020

agilgur5 mentioned this issue Oct 7, 2020

fix/github: typo /king bug -> /kind bug #609

Merged

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 7, 2020

This was referenced Oct 7, 2020

docs: add list of flags to README directly for quick access #610

Closed

github: add spoiler tags to bug report issue template #612

Merged

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 7, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 6, 2021

serathius closed this as completed Feb 21, 2021

agilgur5 mentioned this issue Apr 16, 2023

docs(release): add improved security of metrics-server note to 1.19 kubernetes/kops#15327

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`manager.go` "x509: certificate signed by unknown authority, unable to fully scrape metrics from source" #541

`manager.go` "x509: certificate signed by unknown authority, unable to fully scrape metrics from source" #541

agilgur5 commented Jun 11, 2020 •

edited

Loading

fejta-bot commented Sep 9, 2020

agilgur5 commented Oct 7, 2020

fejta-bot commented Jan 7, 2021

fejta-bot commented Feb 6, 2021

serathius commented Feb 21, 2021

unacceptable commented Apr 7, 2022

manager.go "x509: certificate signed by unknown authority, unable to fully scrape metrics from source" #541

manager.go "x509: certificate signed by unknown authority, unable to fully scrape metrics from source" #541

Comments

agilgur5 commented Jun 11, 2020 • edited Loading

fejta-bot commented Sep 9, 2020

agilgur5 commented Oct 7, 2020

fejta-bot commented Jan 7, 2021

fejta-bot commented Feb 6, 2021

serathius commented Feb 21, 2021

unacceptable commented Apr 7, 2022

`manager.go` "x509: certificate signed by unknown authority, unable to fully scrape metrics from source" #541

`manager.go` "x509: certificate signed by unknown authority, unable to fully scrape metrics from source" #541

agilgur5 commented Jun 11, 2020 •

edited

Loading